Repository logo

Research Matters 02


Recent Submissions

Now showing 1 - 9 of 9
  • ItemOpen AccessPublished version Peer-reviewed
    A cognitive psychological exploration of the GCSE marking process
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Suto, Irenka; Greatorex, Jackie
    GCSEs play a crucial role in secondary education throughout England and Wales, and the process of marking them, which entails extensive human judgement, is a key determinant in the futures of many sixteen-year-olds. The aims of our study were to investigate the cognitive strategies used when marking GCSEs and to interpret them within the context of psychological theories of human judgement. Two GCSE examinations were considered: an intermediate tier Mathematics paper, which used a 'points-based' marking scheme, and a foundation tier Business Studies paper, which used a 'levels-based' scheme. For each subject, a group of six experienced examiners marked four identical script samples each. The first three of these samples were marked silently. Whilst marking the fourth sample, the examiners were asked to 'think aloud' concurrently. Using a semi-structured interview schedule, the examiners were later questioned about their marking experiences retrospectively. A qualitative analysis of the verbal protocol data enabled us to propose a tentative model of marking, which includes five distinct cognitive marking strategies: matching, scanning, evaluating, scrutinising, and no response. These strategies were broadly validated not only in the retrospective interviews with the participating examiners, but also by other senior mathematics and business studies examiners.
  • ItemOpen AccessPublished version Peer-reviewed
    Research Matters 2: June 2006
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Green, Sylvia
    Research Matters is a free biannual publication which allows Cambridge University Press & Assessment to share its assessment research, in a range of fields, with the wider assessment community. 
  • ItemOpen AccessPublished version Peer-reviewed
    What happens when four Financial Times' journalists go under the eye of the invigilator?
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Green, Miranda
    This article details the experiences of four journalists taking an A level or AS level examination paper apiece as an experiment. Only a small amount of preparation was undertaken and the purpose of the experiment was to see how well they would perform based upon their existing knowledge. A detailed description of the journalists' varied emotions is provided as well as their reflections upon the content and standards of the papers in question. The article was published in The Financial Times on Saturday August 20th 2005 and is reproduced in Research Matters with their permission.
  • ItemOpen AccessPublished version Peer-reviewed
    The curious case of the disappearing mathematicians
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Bell, John; Emery, Joanne
    This article shows that an apparently alarming fall in the number of A level mathematics students between 1989 and 2001 can be accounted for by changes in the structure of the mathematics A level leading to candidates dropping out rather than failing, and to demographic changes.
  • ItemOpen AccessPublished version Peer-reviewed
    The Cambridge Assessment/Oxford University automatic marking system: Does it work?
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Raikes, Nick
    In the first issue of Research Matters, Sukkarieh et al. (2005) introduced our work investigating the automatic marking of short, free text answers to examination questions. In this article, I give details and results of an evaluation of the final prototype automatic marking system that was developed.
  • ItemOpen AccessPublished version Peer-reviewed
    Judging learners' work on screen: Issues of validity
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Greatorex, Jackie
    Current developments in Cambridge Assessment and elsewhere include assessors marking digital images of examination scripts on computer, rather than the original scripts on paper, and judges marking and moderating digitally produced coursework on computer, rather than on paper. One question such innovations raise is whether marks from judgements made about the same work presented on computer and on paper are comparable. Generally the literature concerning the on-screen marking of tests and examinations suggests that on-paper and on-screen scores are indeed comparable (e.g., Bennett, 2003; Greatorex, 2004), although Fowles and Adams (2005) report that differences have been found in studies by Whetton and Newton (2002), Sturman and Kispal (2003) and Royal-Dawson (2003). Our concern in this discussion article is that even when double marking studies find high levels of agreement between marks for the same work judged in different modes, issues of validity might be masked. We are thinking of validity in terms of the cognitive processes of the assessor when reaching a judgement, and how well these reflect the judgements that were intended when the assessment was devised.
  • ItemOpen AccessPublished version Peer-reviewed
    International perspectives on vocational education: What can we learn from each other?
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Suto, Irenka; Green, Sylvia
    The broad aim of Vocational Education and Training (VET) is to provide students with the technical skills and knowledge needed to enter the workforce. It exists for an extensive range of subject areas and may be delivered through many different kinds of training institutions and enterprises. Over the past four years, this huge area of education and assessment has received close scrutiny from the Government and others, as part of a broader review of education and skills among 14 to 19 year olds (Tomlinson, 2004; Department for Education and Skills, 2005). Given that this process has resulted in proposals for considerable reform of VET for this age group, we deemed it important to know more about the international context within which they are set. Who does VET affect globally, and what might we learn from the experiences of other countries? The aims of this project, therefore, were to identify and examine two main types of data: (i) on the extent of participation in VET and its associated assessment worldwide; and (ii) relating to key differences in the VET systems of different countries. There were three stages to the project: 1. A review of the quantitative data available. 2. A review of other key literature. 3. A discussion group at an international conference. In this report, we summarise some of the main findings from each stage. We conclude that, in general, there is a paucity of internationally comparable quantitative data relating to vocational education and training.
  • ItemOpen AccessPublished version Peer-reviewed
    I expect to get an A': How the FT writers thought they would do - how they actually did
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Blitz, James; Giles, Chris; Kellaway, Lucy; Lloyd, John
    This article is a continuation of the preceding piece 'What happens when four Financial Times' journalists go under the eye of the invigilator' which describes the reactions of journalists taking an A level or AS level examination paper apiece in a subject related to their career experience - Politics, Business Studies, Economics and Media Studies. This short piece provides details of their results and the markers' comments. The article was published in The Financial Times Weekend in 2005 and is reproduced in Research Matters with their permission.
  • ItemOpen AccessPublished version Peer-reviewed
    Examiners annotations: practice and purpose
    (Research Division, Cambridge University Press & Assessment, 2006-06-01) Crisp, Vicki; Johnson, Martin
    The processes of reading and writing are recognised to be inextricably intertwined. Writing helps to support cognitive demands made upon the reader whilst processing a text (e.g., O'Hara, 1996; Benson, 2001). Examiners annotate scripts whilst marking (e.g., underlining, circling, using abbreviations or making comments) and this may reflect the cognitive support for comprehension building that annotations can provide. There is also some existing evidence that annotations might act as a communicative device in relation to accountability and that annotating might have a positive influence on markers' perceptions and affect their feelings of efficacy. This research investigated the use of annotations during marking and the role that annotations might be playing in the marking process. Six mathematics GCSE examiners and six business studies GCSE examiners who had previously been standardised to mark the paper were recruited. Examiners initially marked ten scripts which were then reviewed by their Team Leader. Examiners then marked a further 46 (Business Studies) or 40 (Mathematics) scripts. The examiners later attended individual meetings with researchers. The session began with each examiner marking a small number of new scripts to re-familiarise themselves with the examination paper and mark scheme. A researcher then observed each examiner as they continued marking a few further scripts. Each examiner was interviewed about their use of annotations. The findings portray a clear sense that markers in both subjects believed that annotating performed two distinct functions. The first appeared to be justificatory, communicating the reasons for their marking decisions to others. This mirrors the statutory requirements for awarding bodies to establish transparent, accountable procedures which ensure quality, consistency, accuracy and fairness. The second purpose was to support their thinking and marking decisions. In addition to helping markers with administrative aspects of marking (for example, keeping a running tally of marks), there are claims that annotations also support higher order reading comprehension processes.