Repository logo
 

Realistic precision and accuracy of online experiment platforms, web browsers, and devices.

Accepted version
Peer-reviewed

No Thumbnail Available

Type

Article

Change log

Authors

Anwyl-Irvine, Alexander  ORCID logo  https://orcid.org/0000-0002-3792-7745
Dalmaijer, Edwin S 
Hodges, Nick 
Evershed, Jo K 

Abstract

Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use.

Description

Keywords

Accuracy, Automated hardware testing, Big data, Experiment builder, MTurk, Online testing, Psychophysics, Reaction time, System testing, Behavioral Research, Data Accuracy, Humans, Internet, Reaction Time, Software, Web Browser

Journal Title

Behav Res Methods

Conference Name

Journal ISSN

1554-351X
1554-3528

Volume Title

53

Publisher

Springer Science and Business Media LLC

Rights

All rights reserved
Sponsorship
Medical Research Council (MC_UU_00005/2)