Randomizing Vignette Factorial Designs in Survey Research with Qualtrics

Last updated on May 23, 2020 4 min read Research

In survey research, we sometimes want to present varying conditions in a short descriptive text, often called a vignette, and measure the effects of those conditions on an outcome of interest.

For example, say we have been hired by DinoCreations Inc. to gauge the public’s willingness to spend tax dollars on our brand new Island Adventure theme park, complete with living dinosaurs created from ancient DNA held within amber! We want to know if varying the dinosaurs type and size in a vignette results in statistically significant differences in potential patrons willingness to visit. We have two types of dinosaurs, and they come in three sizes. Will our potential guests be more likely to purchase a vacation package dependent on the type and size of dinosaurs we advertise with?

There are multiple ways we could do this. For example, we could just type out each of the six vignette (2x3) and have those randomly display to respondents. However, some vignette designs have a lot more conditions to evaluate. Here, I’ll demonstrate a simple 2x3 vignette factorial design, but the method can easily be modified for larger designs. The survey platform Qualtrics makes this easy, using embedded data and the randomizer in survey flows.

Placeholder Descriptive Text

Following months of research, we have determined that there are TWO types of dinosaurs (T-rex and Stegosaurus), and each dinosaur comes in three sizes (big, humongous, and ginormous). Lucky for us, this is a perfect opportunity to put a 2x3 factorial vignette design into play!

For now, let’s create a place holder question in the survey. Our vignette will be about dinosaurs, and we are interested in willingness to visit our new Dinocreations.

Building the Randomizer

Next we need to build a randomizer inside the survey flow. Once inside the survey flow, click the “add below” text inside the survey block, then click the “randomizer” tag.

Now we want to add the type of information this randomizer will handle, in this case, “type” of dinosaur. Click “Add a new element here,” using the “embedded data” type, and assign “type” as the element name. Then assign a type of dinosaur, I chose “terrifying Tyrannasaurus Rex.” Then repeat this step, and assign a different type of dinosaur, in this case an “adorable Stegosaurus.” Finally, make sure you tell the randomizer to randomly present only 1 of these options. Click the “evenly present options” if you want there to be a randomized and equal chance for your survey respondents to see each choice (so here, a 1:2 chance, but for size, a 1:3 chance).

Now we want to build another randomizer, this time to vary the size of the dinosaur in our vignette. Using the same steps as above, we will add a size randomizer that uses three elements of big, humongous, and ginormous, and then make sure only one of these options is presented.

For the final step in the survey flow, we want to make sure both randomizers are ABOVE the vignette question. This assures that the appropriate information is placed into our survey vignette question.

Back to the vignette, and inserting piped text

Save and close the survey flow interface. Now we need to use the “piped text” interface to get our vignette put together. We have decided to concentrate on customers who will be leaving on their honeymoon soon. The plain text in this box is text that EVERY survey respondent will see, while the SIZE and TYPE placeholders will need to be modified to import the embedded data we defined in the survey flow steps above.

To use the piped text funcionality, highlight SIZE, and then click the “piped text” tab directly above:

Then, click the “embedded data” option, and type in “size” (without the quotation marks) and hit enter. Repeat this step for the TYPE placeholder, with the “type” embedded data. After you delete the placeholders, you should have something that looks like:

Now, let’s see if our survey is working. Use the “preview” button at the top of the page. Check it a few times to make sure the randomization is working. Here, in two consecutive previews, I can already see the randomization is working!