Exploring the Validity of Comparative Judgement: Do Judges Attend to Construct-Irrelevant Features?
Authors
Chambers, L
Cunningham, E
Publication Date
2022Journal Title
Frontiers in Education
ISSN
2504-284X
Publisher
Frontiers Media SA
Volume
7
Language
en
Type
Article
This Version
VoR
Metadata
Show full item recordCitation
Chambers, L., & Cunningham, E. (2022). Exploring the Validity of Comparative Judgement: Do Judges Attend to Construct-Irrelevant Features?. Frontiers in Education, 7 https://doi.org/10.3389/feduc.2022.802392
Abstract
<jats:p>When completing a comparative judgment (CJ) exercise, judges are asked to make holistic decisions about the quality of the work they are comparing. A key consideration is the validity of expert judgements. This article details a study where an aspect of validity, whether or not judges are attending to construct-irrelevant features, was investigated. There are a number of potentially construct-irrelevant features indicated in the assessment literature, and we focused on four features: appearance; handwriting; spelling, punctuation, and grammar (SPaG); and missing response vs. incorrect answer. This study explored this through an empirical experiment supplemented by judge observation and survey. The study was conducted within an awarding organisation. The particular context was within a programme of work trialling, a new method of maintaining examination standards involving the comparative judgement of candidates’ examination responses from the same subject from two different years. Judgements in this context are cognitively demanding, and there is a possibility that judges may attend to superficial features of the responses they are comparing. It is, therefore, important to understand how CJ decisions are made and what they are or are not based on so that we can have confidence in judgements and know that any use of them is valid.</jats:p>
Keywords
Education, comparative judgment, standard maintaining, construct-irrelevance, validity, assessment
Identifiers
External DOI: https://doi.org/10.3389/feduc.2022.802392
This record's URL: https://www.repository.cam.ac.uk/handle/1810/337037
Rights
Licence:
http://creativecommons.org/licenses/by/4.0/
Statistics
Total file downloads (since January 2020). For more information on metrics see the
IRUS guide.
Recommended or similar items
The current recommendation prototype on the Apollo Repository will be turned off on 03 February 2023. Although the pilot has been fruitful for both parties, the service provider IKVA is focusing on horizon scanning products and so the recommender service can no longer be supported. We recognise the importance of recommender services in supporting research discovery and are evaluating offerings from other service providers. If you would like to offer feedback on this decision please contact us on: support@repository.cam.ac.uk