In a comment on our blog post, Robin van Emden made us aware of jsPsych, an alternative framework for online experimentation (see his comment here), and asked us how it compared to the QRTEngine. Please note that my findings are based on a necessarily limited, though hopefully representative, experience of working with jsPsych. I have tried to be as honest as possible about both methods, because I feel the jsPsych method in some cases is a serious alternative to the QRTEngine for online experimentation. Prior to publication, we gave Josh de Leeuw, author of jsPsych, the opportunity to review the post and have revised the document based on his comments.

Both our method, the QRTEngine, and the jsPysch attempt to fill up a gap that currently exists in online experimentation. There is a need for a method that provides a flexible framework for building experiments, that is highly precise in each individual run of the experiment (ie, approaches the researcher’s intended experiment with high accuracy on individual machines, not a trivial feat!), is easy to set-up, which makes the data storage and analysis easy and runs natively in browsers (no plug-ins required, preferably not even Flash). Based on my experience of building an experiment, I think that the QRTEngine and jsPsych both have different strengths and weaknesses in the aforementioned needs for a new framework. The following table quickly summarizes those strengths, and weaknesses. Strengths are denoted with a +, weaknesses with a -.

* = Will be implemented in jsPsych v4.0, to be released end of october.

** = Will be implemented in QRTEngine v17.

Please read on for a more in-depth look into why those strengths and weaknesses arise.

Flexibility

First off, the QRTEngine is heavily integrated with Qualtrics (to the point where it would lose a lot of functionality if taken out of this system). jsPsych is more a stand-alone client-side framework that, granted, is a whole lot more flexible than the QRTEngine, due to its independence of any specific (survey) platform. Personally, I like the modularity of the jsPsych framework, where plug-ins are treated as ‘templates’ for trials and trial blocks, and the ‘experimental flow’ is dictated by the order in which these templates are requested. In the QRTEngine, this would be similar to using Qualtrics’ library function to import blocks that implement certain paradigms, at will. However, Qualtrics doesn’t make this convenient at the moment, as you can’t ‘subscribe’ to any library at will, it is either account (which is limited), company (again, limited) or Qualtrics-wide (not everyone who uses Qualtrics is interested in the QRTEngine). On the other hand, as the jsPsych author himself notes, it is currently inconvenient to perform conditional and looping structures (come jsPsych v4.0, this will no longer be true) in your experiment and although QRTEngine doesn’t support loops at the moment (although plans have been made to implement support for those, through forward-skip and back-track logic), the QRTEngine does support a wide array of types of conditional structures, both through Qualtrics’ own interface and the within-trial QRTEngine interface. One major constraint in the QRTEngine still hasn’t been resolved, which is the impossibility of arbitrarily low ITIs. Between-trial page reloads take time, a problem which jsPsych doesn’t suffer from, as full trial blocks can be ran within one page-load. So, both methods have their ups and downs regarding flexibility of the framework, however jsPsych seems to win out on flexibility.

Accuracy

The accuracy of both methods is something where the QRTEngine, at the moment, seems to have an advantage. The architecture of the QRTEngine provides a central loop that not only goes through stimuli in trials (and trials) in sequence, but also directly provides highly accurate timings, through the use of the requestAnimationFrame (rAF) method (and less accurate timings through setTimeout where rAF is not available). rAF is a relatively new JavaScript feature that allows callbacks to be synchronized with the screen refreshes, making sure that the function is called no more than once each screen refresh. Next to that, rAF callbacks have a higher priority than setTimeout callbacks, which means that it is less likely that callbacks are delayed due to other ‘things’ happening in the webpage (keyboard/mouse responses, garbage collection). This all happens within tens of milliseconds, which makes setTimeout infeasible for experiments such as Masked Priming (which we reported on in our paper), but it should be fine for experiments that require accuracies of around 200 ms (or higher).

Currently, jsPsych does not use rAF, which immediately results in less accurate timings (only using setTimeout, which is similar to the method used by Crump, 2013), however an issue ticket has been created in the jsPsych repository by the original author to address this issue, which is set to be resolved by the coming version 4.0. So, as of this writing, if highly accurate timing within a trial is your main concern and you accept bad timing of the inter-trial interval, the QRTEngine seems to be your best choice. After the release of jsPsych v4.0, there should be no more major differences in the accuracy of timing.

Ease-of-use

Ease-of-use is usually a bit at odds with flexibility. On the one hand when creating an experiment, you want as few actions to give you an experiment as powerful as possible, however this usually means abstracting many low-level details away, reducing the flexibility of the framework in the process. I feel like the QRTEngine is slightly easier in use as less of the snippets actually has to be understood for the QRTEngine to set-up the flow of an experiment than for jsPsych, which requires you to string plug-ins together and to set up your own plug-in if the current plug-ins don’t provide a template for the paradigm you need (granted – it should, as the framework becomes more popular, happen less and less). As for creating your stimuli, I feel like the Qualtrics wysiwyg editor has a definite leg-up on jsPsych here. Being able to use Qualtrics’ wysiwyg editor allows you to 1) overview your current trial as you set it up and 2) reduces the technical expertise needed in writing stimulus content in HTML and styling it with CSS. Next to that, the QRTEngine integrates very well with every other aspect of Qualtrics, the resulting data file (after parsing) is in long format with the data gathered through the QRTEngine mixed in seamlessly with data gathered through conventional Qualtrics survey aspects. Thus, what the QRTEngine lacks in flexibility, in a sense, it makes up for in ease-of-use.

Data storage

There are two main parts of the data storage that I’d like to discuss. First off, there’s the within-experiment data storage and its API. jsPsych automatically stores all data related to previous trials and allows you to fairly easily recall all this data (even from several trials back). The QRTEngine’s data storage & API is modeled after E-Prime’s, where one is expected toexplicitly store any relevant information about previous trials in order to be able to use them in following trials. Frankly, I feel like the jsPsych data API and the scope of it is superior to the QRTEngine’s, so one thing I consider working on is extending the scope of the QRTEngine data API, hopefully making it easier in use (without breaking backwards compatibility). On the other hand, the QRTEngine does provides more extensive data reporting out of the box, such as onset and offset times of stimuli, durations (both intended and measured), onset times of responses and more.

Secondly, there’s the issue of aggregating the data of all individual participants into one big data file and then analysing that data. This, apparently, is outside of the scope of jsPsych; one needs a third party platform, as no native method is provided to submit the gathered data back to the server. The author of jsPsych has published a paper integrating jsPsych with psiTurk, which allows for easy data collection on mechanical turk. Robin van Emden has supplied a method to do this in Qualtrics by stuffing the data into an embedded data field. I can’t comment on how easy-to-use psiTurk is, as I haven’t tried it out, but the method Robin van Emden uses makes the jsPsych part of the survey fairly inconsistent ( the jsPsych data is stored in ‘long format’, whereas qualtrics data is stored in ‘wide format’, which would make building the final data file harder by hand) with the remaining part of the Survey. On the other hand, although the QRTEngine’s final data file nicely integrates QRTEngine data with conventional Qualtrics data, the parsing process necessary to get such a data file is one of the major weaknesses in the experimental flow of the QRTEngine. The parsing tool I feel, needs a lot of polish to make it, for example, useable on Linux and to make it more robust against failures (more transparent error logs are in the making). And the process itself (of parsing) is necessarily fairly complex, building the parser is definitely non-trivial. Here it is sort of a toss-up, do I set-up my own easier to use platform for collecting data with jsPsych, or do I use a method that relies on a complex parsing process that is difficult to troubleshoot if it fails.

Other points

A last point that has to be specifically mentioned is that the QRTEngine relies on Qualtrics for it to work. Although free Qualtrics trials are available, a full-blown license can be fairly expensive. Setting up your own web-server and running a jsPsych experiment is probably much cheaper nowadays, but it requires more technical expertise. As Qualtrics is in use by many universities for normal Surveys anyway, the issues become a bit less prominent.

Conclusion

If we summarize the previously mentioned points, we can conclude that each method has its own strengths and weakness. The jsPsych framework currently is better in terms of flexibility and within-experiment data storage, whereas I feel the QRTEngine is easier to use, both in designing your experiments (within what is allowed by the framework) and extracting a useable data file from the platform the experiment is ran on (even though it makes use of a complex parsing process to do so). Also, due to the QRTEngine using requestAnimationFrame to present stimuli, it is currently more accurate in terms of timings, and has superior reporting on these timings. The most accurate version of the QRTEngine is less well supported by older browsers, but it rolls back to an equally accurate version to jsPsych, if rAF is not supported by the browser.

Let’s conclude by mentioning that both jsPsych and the QRTEngine have a few steps to make when it comes to their development. However, currently, the QRTEngine requires less of a technical expertise from the researcher than jsPsych does. jsPsych, on the other hand, is more flexible in terms of which types of paradigms you can build, if you know a bit about programming (in JavaScript). Furthermore, the QRTEngine seems to provide higher precision timing than jsPsych does and more extensive reporting on those timings. If the highest accuracy is required, the QRTEngine would currently be the prefered method for running those experiments, but only if arbitrarily low ITIs are not necessary. If the highest accuracy is not a requirement, users interested in online experimentation should consider the remaining requirements of their current experimental paradigm, and choose whichever method fits best with those requirements, as both jsPsych and the QRTEngine have their strengths and weaknesses.

Note: jsPsych v4.0 is set to be released soon (end of October) and brings significant fixes to issues listed in this blog-post. I will update the post once I have a chance to test out this new version.