Research Matters 06


Recent Submissions

Now showing 1 - 9 of 9
  • ItemPublished versionOpen Access
    Research Matters 6: June 2008
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Green, Sylvia
    Research Matters is a free biannual publication which allows Cambridge University Press & Assessment to share its assessment research, in a range of fields, with the wider assessment community. 
  • ItemPublished versionOpen Access
    The future of assessment - the next 150 years?
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Oates, Tim
    Prediction is very difficult, particularly if it's about the future' - Niels Bohr. This article examines the theoretical and practical issues surrounding prediction of future approaches to assessment as well as describing areas of likely development - and the form which assessment might take. It uses theory from sociology, philosophy of science and public policy analysis to examine the possibilities of prediction. It then examines emergent areas of development and interest, discussing issues, tensions and possibilities. The specific areas covered comprise: adaptive testing; authentic tasks; on demand testing; influence of technology; outcomes-based learning; teacher assessment in high accountability settings; tiered examinations; 'levels' and grades; and maintenance of standards and measurement error.
  • ItemPublished versionOpen Access
    Marking essays on screen: towards an understanding of examiner assessment behaviour
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Shaw, Stuart
    Computer assisted assessment offers many benefits over traditional paper methods. In translating from one medium to another, however, it is crucial to ascertain the extent to which the new medium may alter the nature of the assessment and marking reliability. Appropriate validation studies must be conducted before a new approach can be implemented in high-stakes contexts. The pilot described here is the first attempt by Cambridge Assessment International Education (formerly Cambridge International Examinations) to mark on-screen extended stretches of written text for the Cambridge Checkpoint English Examination. The pilot attempts to investigate marker reliability, construct validity and whether factors such as annotation and navigation differentially influence marker performance across the on-paper and on-screen marking modes. Candidates wrote their answers on paper scripts in the normal way. The scripts were then scanned and digital images of them were sent by secure electronic link to examiners for on-screen marking at home using Scoris(r) software. It can be relatively hard for examiners to make a full range of annotations when marking on screen. For this reason annotation sophistication was manipulated in the pilot as well as marking mode. Four marking methods were compared: on-paper with sophisticated annotations (current practice); on-paper with simplified annotations; on-screen with sophisticated annotations; and on-screen with simplified annotations. The pilot found that paper-based and screen-based inter-examiner reliability is high for the Cambridge Checkpoint English Examination. Although inter-rater reliability is lower on-screen, it is only marginally deflated.
  • ItemPublished versionOpen Access
    Holistic judgement of a borderline vocationally-related portfolio: a study of some influencing factors
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Johnson, Martin
    The assessment of a large portfolio of mainly textual evidence demands an assessor to accommodate a great deal of information. This comprehension process is influenced by the linear nature of the reading process which leads to the gradual construction of a mental representation of the text in the head of the reader (Johnson and Laird, 1983). Understanding how assessors work with portfolios also requires us to consider how assessors integrate and combine different aspects of an holistic performance into a final judgement. Sanderson (2001) suggests that the social context of the assessor is important to consider since it recognises their participation in a community of practice (Wenger, 1998) and constitutes an 'outer frame' for their activity. This study sought to explore issues of consistent assessor judgement by gathering data about individual assessors' cognitive activity as well as the socio-contextual features in which their practices were undertaken. It focused on an OCR Nationals unit in Health and Social Care (Level 2). Six assessors were asked to 'think aloud' whilst they judged the unit. This commentary was then transcribed into a verbal protocol and analysed with qualitative text analysis software. A modified Kelly's Repertory Grid (KRG) interview technique was also used to gather data about different assessors' perceptions of constructs within the same assessment criteria.
  • ItemPublished versionOpen Access
    Do assessors pay attention to appropriate features of student work when making assessment judgements?
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Crisp, Vicki
    It is via the judgements of appropriate experts that assessment decisions are made, yet the actual thought processes involved during marking or grading are under-researched. This article draws on a study of the cognitive and socially-influenced processes involved in marking and grading A level geography examinations and pilot research into the marking of GCSE coursework by teachers. This data was used to investigate whether assessors pay attention to appropriate features of student work. Verbal protocols of assessors' thinking aloud whilst marking and grading work were collected and measures of marker agreement were obtained. The protocols were analysed in detail using appropriate coding schemes. From the behaviours identified, a tentative model of the marking process was developed, within which features of student work affecting judgements and social and personal reactions were identified. Whilst many features that appeared to influence evaluations were clearly focussed on the criteria intended for evaluation, some were not and could have influenced evaluations. Reactions to language use or legibility (when not assessing communication), personal or emotional responses and social responses sometimes occurred before marking decisions. The article discusses whether such responses could explain variations in marks from different examiners.
  • ItemPublished versionOpen Access
    Critical Thinking - a definition and taxonomy for Cambridge Assessment
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Black, Beth; Chislett, Joe; Thompson, Annne; Thwaites, Geoff; Thwaites, Jacquie
    There are a vast number of Critical Thinking definitions in the literature, which are highly varied and often multi-faceted. The construct of Critical Thinking is hotly debated, with a number of key battlegrounds. The implications of such differing conceptions reach out beyond academic journals. They impact upon educationalists in a number of practical ways, such as devising the best training or delivery model for Critical Thinking; designing and delivering valid assessments which are authentic and which nurture good Critical Thinking skills in students. The main aim of this research was to create a Cambridge Assessment definition and taxonomy for Critical Thinking.
  • ItemPublished versionOpen Access
    Cookery examined - 1937-2007: Evidence from examination questions of the development of a subjects over time
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Elliott, Gill
    This paper describes the evolution of a subject - cookery, later known as home economics and food technology - over time, as seen from the perspective of examination questions. The historical background to the examinations is explored, and examples given from examination questions through the years.
  • ItemPublished versionOpen Access
    Annotating to comprehend: a marginalised activity?
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Johnson, Martin; Shaw, Stuart
    One of the important premises underlying this article is that the cognitive processes involved in reading can play a significant role in assessment judgements. Although we acknowledge that not all assessments of performance rely on assessors appraising written texts, many tests use written evidence as an indicator of performance. As a result, it is important to consider the role of assessors' comprehension building when reading candidates' textual responses, particularly where candidates are offered a greater freedom in determining the form and scope of their responses. This paper brings together literature about linguistics and annotation practices, both empirical and theoretical, and suggests that a critical link exists between annotating and reading activities. Through making the different functions of annotation explicit the intention of this paper is to primarily amplify the importance of the impact of annotating on assessor comprehension.
  • ItemPublished versionOpen Access
    3Rs' of assessment research: Respect, Relationships and Responsibility – what do they have to do with research methods?
    (Research Division, Cambridge University Press & Assessment, 2008-06-01) Johnson, Martin
    This article focuses on the merits and challenges of using qualitative research methods, and how these can contribute positively to the study of assessment. Based on a presentation from a methods-related research seminar, this paper appeals for research engagement with those areas where assessment affects the lives of others. This appeal means not only asking the difficult questions but also having the appropriate methodologies to try to answer them. The paper goes on to champion the strengths of mixed methods approaches that allow for triangulation, complementarity (where findings gained through one method offer insights into other findings) and expansion (of the breadth and scope of the research beyond initial findings).