Research Matters 11
Permanent URI for this collection
Browse
Recent Submissions
Item Open Access Published version Peer-reviewed Research Matters 11: January 2011(Research Division, Cambridge University Press & Assessment, 2011-01-01) Green, SylviaResearch Matters is a free biannual publication which allows Cambridge University Press & Assessment to share its assessment research, in a range of fields, with the wider assessment community.Item Open Access Published version Peer-reviewed Tracing the evolution of validity in educational measurement: past issues and contemporary challenges(Research Division, Cambridge University Press & Assessment, 2011-01-01) Shaw, Stuart; Crisp, VickiValidity is not a simple concept in the context of educational measurement. Measuring the traits or attributes that a student has learnt during a course is not like measuring an objective property such as length or weight; measuring educational achievement is less direct. Yet, educational outcomes can have high stakes in terms of consequences (e.g., affecting access to further education), thus the validity of assessments is highly important. Tracing this trajectory of evolution, particularly through key documents such as the validity/validation chapter in editions of Educational Measurement (Cureton, 1951; Cronbach, 1971; Messick, 1989; Kane, 2006) and the Standards of Educational and Psychological Testing (AERA, APA and NCME, 1954/1955, 1966, 1974, 1985, 1999) has been important to us as part of work to develop an approach to validation for general assessments. The concept of validity is not a new one. Conceptualisations of validity are apparent in the literature from around the turn of the twentieth century, and since that time, they have evolved significantly. Earliest perceptions of validity were that of a static property captured by a single statistic, usually an index of the correlation of test scores with some criterion (Binet, 1905; Pearson, 1896; Binet and Henri, 1899; Spearman, 1904). Through various re-conceptualisations, contemporary validity theory generally sees validity as about the appropriateness of the inferences and uses made from assessment outcomes, including some consideration of the consequences of test score use. This article traces the progress and changes in the theorisation of validity over time and the issues that led to these changes.Item Open Access Published version Peer-reviewed The effects of GCSE modularisation: a comparison between modular and linear examinations in secondary education(Research Division, Cambridge University Press & Assessment, 2011-01-01) Vidal Rodeiro, Carmen; Nadas, RitaAs part of a recent reform in education, the assessment of GCSEs is now organised into modules which can either be taken at the end of the course in a linear fashion, or can be taken throughout the course in a modular approach to teaching and learning. Previous research suggested that students who are assessed early in the course are disadvantaged by their immaturity, if not by their narrower experience of the subject and perform worse than those assessed at the end. Also, it has been argued that regular feedback on performance helps to identify learning needs and encourages students to do better. The present study set out to investigate the above claims combining quantitative and qualitative research. In the quantitative strand of the research, the performances of English and mathematics students were analysed. In the qualitative strand, questionnaires and interviews with students and teachers of both subjects were conducted.Item Open Access Published version Peer-reviewed Rank ordering and paired comparisons - the way Cambridge Assessment is using them in operational and experimental work(Research Division, Cambridge University Press & Assessment, 2011-01-01) Bramley, Tom; Oates, TimIn this article we describe the method of paired comparisons and its close relative, rank-ordering. Despite early origins, these scaling methods have been introduced into the world of assessment relatively recently, and have the potential to lead to exciting innovations in several aspects of the assessment process. Cambridge Assessment has been at the forefront of these developments and here we summarise the current 'state of play'.Item Open Access Published version Peer-reviewed Extended essay marking on screen: Does marking mode influence marking outcomes and processes?(Research Division, Cambridge University Press & Assessment, 2011-01-01) Shiell, Hannah; Johnson, Nat; Hopkin, Rebecca; Nadas, Rita; Bell, JohnResearch into comparisons between how people read texts on paper and computer screen suggests that the medium in which a text is read might influence the way that a reader comprehends that text. This is because some of the reading behaviours that support comprehension building, such as seamless navigation and annotation of text, are not easily replicated on screen. Additional research also suggests that reading long texts can be more cognitively demanding on screen, and that this extra demand can have a detrimental effect on how readers comprehend longer texts. In the context of examination marking, there might be concerns that such a mode-related effect might lead to essays being marked less accurately when marked on screen compared with when they are marked on paper. To investigate further the potential links between marking mode and the outcomes and processes of extended essay marking, the current project replicated an earlier study (Johnson and Nadas, 2009), replacing GCSE essays with longer Advanced GCE essays. The current project considered three broad areas of enquiry, exploring mode-related influences on (i) marking outcomes, (ii) manual marking processes and (iii) cognitive marking processes.Item Open Access Published version Peer-reviewed Does doing a critical thinking A level confer any advantage for candidates in their perfomance on other A levels?(Research Division, Cambridge University Press & Assessment, 2011-01-01) Black, Beth; Gill, TimCritical Thinking AS level was introduced in schools in 2001. There is much research that shows that the teaching of Critical Thinking (CT) does improve critical thinking skills. However, there is less research which shows whether CT skills can be profitably transferred to other subject domains. The purpose of this research was to investigate whether taking an AS level in CT improves performance in other A levels. Using national examination data from 2005 and 2006 we compared the performance of CT students with those not taking CT. The results of both a basic comparison of mean A level performance and a regression analysis (which accounts for prior attainment) showed a significant and positive effect of taking CT, compared with not doing so. According to the regression results, the difference was equivalent to about one tenth of a grade per A level, although the difference was greater for those achieving a higher CT grade.Item Open Access Published version Peer-reviewed Comparing the demand of syllabus content in the context of vocational qualifications : literature, theory and method(Research Division, Cambridge University Press & Assessment, 2011-01-01) Novakovic, Nadezda; Greatorex, JackieOur literature review considers the methods used in studies comparing the demands of vocational syllabus content in England. Generally, categories of demands are either derived from subject experts' views or devised by researchers. Subsequently, subject experts rate each syllabus on each demand category and comparisons can be made. However, problems with the methods include: i) Some studies over-focus on the cognitive domain rather than the affective, interpersonal and psychomotor domains; ii) Experts vary in their interpretations of rating scales. Therefore, we suggest creating a framework of demands which includes all four domains, based on a variety of subject experts' views of demands. The subject experts might rank each syllabus on each type of demand, thus avoiding the problem(s) of rating scales, and facilitating comparisons between syllabuses.Item Open Access Published version Peer-reviewed A better approach to regulating qualification standards(Research Division, Cambridge University Press & Assessment, 2011-01-01) Steinberg, Bene't; Hyder, SarahResponding to the Government's 2010 Education White Paper, this article set out four ways in which it was thought standards could best be maintained. First, users need to take the major role in specifying the content criteria of qualifications - enabling them to help set the standards. Secondly, exam boards need to agree between themselves on design criteria - enabling them to set and maintain the standard in relation to each other. Thirdly, 'communities of practice' need to be set up around each qualification - enabling the standards of each qualification to be owned and maintained by all those with a direct interest in them. Lastly, the regulator must focus on standards alone rather than its other current objectives. Its role in this system would be to underpin inter-exam board agreements as well as those between boards and users.