Realistic precision and accuracy of online experiment platforms, web browsers, and devices.


No Thumbnail Available
Type
Article
Change log
Authors
Anwyl-Irvine, Alexander  ORCID logo  https://orcid.org/0000-0002-3792-7745
Dalmaijer, Edwin S 
Hodges, Nick 
Evershed, Jo K 
Abstract

Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use.

Description
Keywords
Accuracy, Automated hardware testing, Big data, Experiment builder, MTurk, Online testing, Psychophysics, Reaction time, System testing, Behavioral Research, Data Accuracy, Humans, Internet, Reaction Time, Software, Web Browser
Journal Title
Behav Res Methods
Conference Name
Journal ISSN
1554-351X
1554-3528
Volume Title
53
Publisher
Springer Science and Business Media LLC
Rights
All rights reserved
Sponsorship
Medical Research Council (MC_UU_00005/2)