Show simple item record

dc.contributor.authorSaravanos, A
dc.contributor.authorZervoudakis, S
dc.contributor.authorZheng, D
dc.contributor.authorStott, Neil
dc.contributor.authorHawryluk, B
dc.contributor.authorDelfino, D
dc.date.accessioned2021-12-09T00:31:31Z
dc.date.available2021-12-09T00:31:31Z
dc.date.issued2021
dc.identifier.isbn978-3-030-90237-7
dc.identifier.issn0302-9743
dc.identifier.urihttps://www.repository.cam.ac.uk/handle/1810/331290
dc.description.abstractIn this study, we investigate the attentiveness exhibited by participants sourced through Amazon Mechanical Turk (MTurk), thereby discovering a significant level of inattentiveness amongst the platform’s top crowd workers (those classified as ‘Master’, with an ‘Approval Rate’ of 98% or more, and a ‘Number of HITS approved’ value of 1,000 or more). A total of 564 individuals from the United States participated in our experiment. They were asked to read a vignette outlining one of four hypothetical technology products and then complete a related survey. Three forms of attention check (logic, honesty, and time) were used to assess attentiveness. Through this experiment we determined that a total of 126 (22.3%) participants failed at least one of the three forms of attention check, with most (94) failing the honesty check – followed by the logic check (31), and the time check (27). Thus, we established that significant levels of inattentiveness exist even among the most elite MTurk workers. The study concludes by reaffirming the need for multiple forms of carefully crafted attention checks, irrespective of whether participant quality is presumed to be high according to MTurk criteria such as ‘Master’, ‘Approval Rate’, and ‘Number of HITS approved’. Furthermore, we propose that researchers adjust their proposals to account for the effort and costs required to address participant inattentiveness.
dc.publisherSpringer International Publishing
dc.rightsAttribution 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.titleThe hidden cost of using Amazon Mechanical Turk for research
dc.typeConference Object
dc.publisher.departmentJudge Business School
dc.date.updated2021-12-07T13:31:53Z
prism.endingPage164
prism.publicationDate2021
prism.publicationNameHCI International 2021 - Late Breaking Papers: Design and User Experience. HCII 2021. Lecture Notes in Computer Science
prism.startingPage147
prism.volume13094
dc.identifier.doi10.17863/CAM.78737
dcterms.dateAccepted2021-05-12
rioxxterms.versionofrecord10.1007/978-3-030-90238-4_12
rioxxterms.versionVoR
dc.identifier.eissn1611-3349
dc.publisher.urlhttps://link.springer.com/chapter/10.1007/978-3-030-90238-4_12
cam.issuedOnline2021-11-20
pubs.conference-nameInternational Conference on Human-Computer Interaction
cam.depositDate2021-12-07
pubs.licence-identifierapollo-deposit-licence-2-1
pubs.licence-display-nameApollo Repository Deposit Licence Agreement


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record

Attribution 4.0 International
Except where otherwise noted, this item's licence is described as Attribution 4.0 International