Research Matters 09


Recent Submissions

Now showing 1 - 6 of 6
  • ItemPublished versionOpen Access
    Research Matters 9: January 2010
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Green, Sylvia
    Research Matters is a free biannual publication which allows Cambridge University Press & Assessment to share its assessment research, in a range of fields, with the wider assessment community. 
  • ItemPublished versionOpen Access
    The reliabilities of three potential methods of capturing expert judgement in determining grade boundaries
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Novakovic, Nadezda; Suto, Irenka
    In England there is a strong public expectation that qualification standards should remain constant over time. At each examination session, awarding bodies must therefore determine the grade boundaries for their examinations that equate to those of previous sessions. We investigated the reliabilities of three methods for capturing the expert judgement of professional examiners who are responsible for maintaining year-on-year examination standards. The methods were those used in: traditional (current) awarding; Thurstone pairs; and rank ordering. In the context of setting grade boundaries in AS level Biology and GCSE English, we conducted a three-way comparison of the intra-method and inter-method reliabilities of the three methods. For each subject, three mutually exclusive sets of examination scripts were created, which were matched for mark. Three groups of ten 'judges' (examiners, matched for experience of the methods) made judgements using each of the three methods on a different set of scripts. It was found that for both subjects, the traditional awarding and Thurstone pairs methods generated very similar boundary marks, except for at the biology A/B grade boundary. The boundary marks generated by rank ordering were all on the lenient side for biology, whereas for the English C/D grade boundary, they were on the severe side.
  • ItemPublished versionOpen Access
    The evolution of international History examinations: an analysis of History question papers for 16 year olds from 1858 to the present
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Shaw, Stuart; Cooke, Gillian
    The focus of this article is on international History examinations for 16 year olds from 1858 to the present day and examines the historical/cultural context for, and the outworking of, the setting of these examinations in the medium of English. Specific reference points throughout this period have been taken and a linguistic analysis applied to the question papers. A variety of archive material has been used to show more general developmental changes to the curriculum throughout the period. The article examines the language used, the candidate base, the regional differences of the papers and the examiner expectations. To put these findings into context, other sources, including examination regulations, examiners' reports and subject committee papers have also been studied. In 1858 when the Cambridge Local Examinations were introduced, history was a compulsory element of the Junior examination. Candidates had to pass in a whole range of subjects to gain a school leaving certificate and English history could not be avoided. 150 years later there is no doubt that school examinations for 16 year olds have undergone radical transformation and for history examinations to have remained unchanged would be unthinkable. The interest lies not in the fact that the examinations changed but in the way they have changed. While the trend is inevitably towards a more familiar, contemporary style, this study also shows that the pace and particular directions of change have been of a less predictable nature.
  • ItemPublished versionOpen Access
    'Key discriminators' and the use of item level data in reporting
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Bramley, Tom
    As more examination papers in general qualifications (GCSEs and A levels) are scanned and marked on screen, the marks on individual questions or question parts are collected automatically, and are referred to as item level data (ILD).The analysis of ILD is available for use in awarding meetings (where the grade boundaries are decided). This article discusses the theoretical rationale for using ILD in awarding, presents some possible formats for displaying data, and suggests ways in which the data could be used in practice.
  • ItemPublished versionOpen Access
    How do examiners make judgements about standards? Some insights from a qualitative analysis
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Greatorex, Jackie
    There is a good deal of research about how judgements are made in awarding when A level and GCSE grade boundaries are chosen. There is less research about how judgements are made in Thurstone paired comparisons and rank ordering (popular methods in comparability studies to compare grading standards). Therefore, the research question for the present study is 'how do Principal Examiners (PEs) make judgements about standards in awarding, Thurstone paired comparisons and rank ordering?' The present article draws from a wider project in which Principal Examiners thought aloud whilst making judgements about the quality of candidates' work and grading standards in awarding, Thurstone paired comparisons and rank ordering situations analogous to how these methods are practised. For the present analysis a coding frame was developed to qualitatively analyse the think aloud data. The coding frame constituted codes grounded in the think aloud data and grade descriptors from the qualification specification. It was found that overall the Principal Examiners attended to valid factors such as where marks were gained, responses to key questions and characteristics of candidates' work that were in the grade descriptors.When the importance of each factor was considered there were some similarities and some differences between the methods. Implications and recommendations are discussed.
  • ItemPublished versionOpen Access
    Exploring non-standard English amongst teenagers
    (Research Division, Cambridge University Press & Assessment, 2010-01-01) Black, Beth
    The main aim of this research was to measure the levels of awareness of non-standard English (NSE) amongst GCSE level students. Through use of a questionnaire/assessment instrument, it sought to add to the empirical research on non-standard written English in young people at GCSE level in the following areas: - which NSE forms were most and/or least recognised; - whether respondents could produce Standard English versions of the NSE forms; - whether respondents could spontaneously use the term non-standard English when asked to identify the type of English used in the assessment instrument; - perceptions of NSE; - whether characteristics of respondents (gender, school type, region) produced any differences in recognition and production of NSE.