Countering the effect of the order of presentation on experimental results is often done by a technique called ‘counterbalancing’. Participants are assigned to two different groups where each group has parts of the experiment displayed in a different order. In this tutorial we will explain how to do counterbalancing in Qualtrics.

Counterbalancing in Qualtrics is surprisingly easily added to our QRTEngine-powered experiments by using the Survey Flow randomizer element, combined with piping Embedded Data into your actual Survey. If these terms seem foreign to you, don’t worry, we will explain them as we introduce them in the tutorial. Please download the following example task: Tutorial – Counterbalancing, as we will refer back to this task occasionally throughout the tutorial. We will be aiming to explain between-subject counterbalancing on a Survey level.

The randomizer element

We started off with the basic Skeleton Task, imported it, opened the Survey Flow and added a randomizer element to the Survey Flow, as shown in the picture below.

The Randomizer is a special Survey Flow element will randomly pick any element that is contained below it. The feature that makes it important is the possibility to present every element an event amount of times. If you make it select only 1 element out of all the possible element contained below it, you will have a skeleton for any counterbalancing you could possible want to do.

Say, you’re running an experiment where you’re interested in the efffect of subjects being rewarded in Euros, rather than Dollars, on their performance on the standard skeleton task we have implemented for the QRTEngine. We will implement this only superficially, by showing some subjects they received a reward in euros upon correctly responding to the skeleton task stimuli; whereas others will be shown that they received their  reward in dollars.

First off, we go back to the Survey Flow, and add two embedded data elements to the Randomizer element we just created. We set a variable called ‘Currency’ in each one, where the value of one will be ‘euros’ and the other will be ‘dollars’, like so:

Furthermore, rather than the default 2, we set the randomizer to only display 1 of the ‘following’ elements per Survey run. This, combined with checking the ‘Evenly Present Elements’ box, will tell Qualtrics to ‘run’ each of the Embedded Data blocks for 50% of the subjects, which will mean that for 50% the Currency is in euros, and the other 50% has the currency in dollars.

Embedded Data

Embedded Data is the medium for storing any data inside the Survey that you might need at a different point in the same Survey. Qualtrics provides powerful tools to further customize the display of your Survey based on the value of these Embedded Data fields. You can choose to display certain questions based on logic involving Embedded Data, or choose not to display certain blocks of questions based on it. It is also possible to ‘Pipe’ in the value of an Embedded Data field into the text or JavaScript of your Survey. The QRTEngine, for example, relies on Embedded Data to have properly functioning data storage capacities. For a more thorough explanation, please see: http://www.qualtrics.com/university/researchsuite/advanced-building/survey-flow/embedded-data/.

Note for the Advanced Users: Qualtrics also provides a half-working JavaScript API to interact with this Embedded Data. Both a set function (Qualtrics.SurveyEngine.setEmbeddedData) and a get function (Qualtrics.SurveyEngine.getEmbeddedData) have been defined, but only the set function works. Getting Embedded Data into your JS variables can only be done through the piped text functionality, which limits your use of embedded data to static references (which, for the QRTEngine, in turn, partially lead to the complex set-up process in the QRTE.Init questions).

We use the ‘Correct’ stimulus in the Skeleton Task to display that they also received 5 of whatever currency they have been assigned to. The way to do this is shown below:

 

Which contains the following text:

${e://Field/Currency} is a reference to the embedded data field Currency. Qualtrics, before this Survey is sent to a subject’s browser, replaces all these references with the actual value contained in the field that is referenced. The subject, even when looking at the HTML source of the page, will never see the reference to the Currency field, only either euros, or dollars.

All that is left is to store which group the user belongs to. We can do a similar trick as we did in the stimulus, but only now in a line of JavaScript code. In the question tagged as Init we have added the marked line of code, which will store the actual value of Currency, every other line has stayed the same:

Changing the order of blocks

Now – rather than euros vs dollars being a between-subject manipulation, we would like the manipulation to be a within-subject. Meaning, we present the same task twice, but once with euros as a reward, and once with dollars as a reward. Doing so, we theoretically could run into the problem of the order-effect, which has been mentioned in the introduction. If euros are consistently presented first to a participant, than it could be that not euros, but the fact that its present first, is the main part of any hypothetical effect that can be found. This is a possible confound and needs to be corrected for. We do this through randomizing the order in which the blocks are presented.

First off, we copy the current skeleton task block we have and give it a unique block Id and correctly set-up the reference to the Exit question, as explained in the ‘copying blocks tutorial’ <insert link>.

Since Randomizer elements only select single elements contained in them, you can’t directly add the two trial blocks to the randomizer element, like this:

 

Why this is wrong is fairly easy to explain. Even though you set the randomizer to display both elements in a random order, there is no guarantee both orders (Block 1 first, 2 second, or Block 2 first, 1 second) are presented to subjects evenly, there is only a guarantee that each individual block is presented evenly. Fair enough, with a perfect random number generator, and an infinite amount of runs, the amount of people experiencing each order will be divided 50/50, however Qualtrics offers us a way to achieve the same without using this limit approach argument. See the following:

 

Qualtrics allows us to randomize over Survey Flow elements, so what we have done is make our block ordering a survey element. A ‘branch element’ is an element that checks a conditional statement, and if it turns out to be true – the contained elements are executed in order. Our conditional always is true, although alwaystrue has not be defined, we check if it is not false (which is not, because it probably is empty === ”). This is the case, so the blocks are in either one of the desired orders. ‘Evenly Present Elements’ will then ensure that each order is presented to subject evenly.

In order to keep the image small, I have excluded the embedded data element that sets the value of Currency. In the actual task download, you will find it set-up properly in the Survey Flow.

Conclusion

We hope that you have a better idea of how to do counterbalancing now. Although this tutorial touches upon the basics, many more advanced application have been devised, for example we have counterbalanced the correspondence between ‘actions’ and the identity of the keys that have to be pressed. We would be interested in hearing about other applications of counterbalancing techniques in the QRTEngine, or simply about any issues that you might have had implementing counterbalance techniques. As always, please feel free to drop us a line at support@qrtengine.com!