The effect of manipulating features of examinees' scripts on their perceived quality

Abstract

Expert judgment of the quality of examinees' work plays an important part in standard setting, standard maintaining, and monitoring of comparability. In order to understand and validate methods that rely on expert judgment, it is necessary to know what features of examinees' work influence the experts' judgments. The controlled experiment reported here investigated the effect of changing four features of scripts from a GCSE Chemistry examination: i) the quality of written English; ii) the proportion of missing as opposed to incorrect responses; iii) the profile of marks in terms of fit to the Rasch model; and iv) the proportion of marks gained on the subset of questions testing 'good Chemistry'. Expert judges ranked scripts in terms of perceived quality. There were two versions of each script, an original version and a manipulated version (with the same total mark) where one of the four features had been altered. The largest effect was obtained by a combination of iii) and iv): increasing the proportion of marks gained on 'good Chemistry' items, and increasing the number of correct answers to difficult questions at the expense of wrong answers to easy questions. The implications of the findings for operational standard maintaining procedures are discussed.