Finishing up your Experiment

During our validation studies for the QRTEngine, we developed several techniques that optimised our experiments. Ranging from connection speed filtering, to pre-loading images in cache. In this tutorial we will explain a few of these tricks.

Adding Text-only Questions

In order to counter-act possible ‘trial flashes’ (github issue), part of the CSS included in the QRTEngine hides every Question by default, after which the QRTEngine systematically ‘unhides’ these questions when the time is right. Sadly, we cannot differentiate between ‘QRTEngine’ questions and ‘Non-QRTEngine’ questions, but considering the gravity of the issue, we decided to opt for hiding every question on pageload. In order to show a page without invoking the QRTEngine, we have added a convenience function that, if called in JavaScript, will unhide the current Question for you.

Please add the following to the JavaScript of the question you’d like to be unhidden:

 

Advance upon key-press

Instead of requiring the participant to click a next button, you sometimes want them to press a key, but again without invoking the QRTEngine. We have added a convenience function for this as well called waitForKey. If you’d like continue upon the participant pressing spacebar, add the following code to the question’s JavaScript:

‘ ‘ stands for spacebar, can be any other key such as ‘C’, or ‘N’, etc.

Pre-loading images in cache

Browsers are able to cache certain parts of a website (JavaScript files, CSS files, images, etc.) on a users hard-drive, so that, in case these files are used multiple times, they don’t need to be re-downloaded, costing lots of bandwidth and time, but can be quickly loaded from the user’s hard-drive.

Behind the scenes, one of things that takes up a lot of time while the QRTEngine runs through a trial, is the loading of that trial. If many elements need to be loaded, the time it takes before the QRTEngine is ready to go is increased, and thus the minimum delay between trials is increased. At least for images, by using the browser’s caching capabilities, we can reduce this initial loading time.

By adding all images to a single question, using HTML like this (replacing the url1 & url2 etc, with actual url’s to the images)

 

and adding the following JavaScript to hide it:

we can create a question that isn’t shown, but allows the browser to load each image that is used in the trial, so that it can simply use the cached versions when the trial is ran, optimising loading times.

Filtering participants based on connection speed

This is a slightly more complex trick. While we were running the validation study we noticed that some people had extremely long delays between trials. We attributed this to slow connections, simply because page-loads seemed to, for some participants, take 8 seconds on average. This would have unwanted effects on our data, so we built a way to filter and block these participants from doing our experiment and in such a way, that they (hopefully) wouldn’t be able to tell.

This is our connection test in action: https://eu.qualtrics.com/SE/?SID=SV_4YMyoCJiFUm6CGh&Preview=Survey

In short, we use a progress bar that repeatedly fills itself as the page is reloaded. The delay between these page loads is measured and after 10 page loads the mean and standard deviation are calculated over these delays. If the likely delay (Mean + 2 * standard deviation) is too high (over 2000 ms in our case), we send the participant down a conditional branch that terminates the survey before they get to run through the trials section.

The example .qsf can be downloaded here.