Repository logo
 

Validity and reliability of automatically generated propositional reasoning Items: a multilingual study of the challenges of verbal item generation

Accepted version
Peer-reviewed

Type

Article

Change log

Authors

Gühne, D 
Doebler, P 
Condon, DM 
Luo, F 

Abstract

This study introduces a newly developed public-domain multilingual automatic item generator that creates propositional reasoning (PR) items belonging to 15 item families by using various inference rules. Psychometric properties of the resulting written PR test were investigated in three diverse samples in English, simplified Chinese, and German, respectively. Internal consistency was good to excellent across samples. The ICAR16 short form test of cognitive abilities (Condon & Revelle, 2014) was used to evaluate construct validity. Correlations of ICAR16 scores and PR scores were high. Furthermore, items within families appeared to be equivalent, with only minor differential item functioning between the Chinese- and English-speaking samples. Performance on the PR test was shown to be reasonably stable over the course of 1 week. Differences of total scores between test forms (pen and paper vs. computerized administration) were not detected. Findings suggest that the automatically generated PR test is a valuable instrument for the assessment of propositional reasoning ability.

Description

Keywords

automatic item generation, propositional reasoning, cognitive ability, cross-cultural research

Journal Title

European Journal of Psychological Assessment

Conference Name

Journal ISSN

1015-5759
2151-2426

Volume Title

37

Publisher

Hogrefe

Rights

All rights reserved
Sponsorship
Economic and Social Research Council (ES/L016591/1)
ESRC (ES/L016591/1)