Repository logo
 

The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data.


Change log

Authors

Wood, Angela M 
Royston, Patrick 
White, Ian R 

Abstract

Multiple imputation can be used as a tool in the process of constructing prediction models in medical and epidemiological studies with missing covariate values. Such models can be used to make predictions for model performance assessment, but the task is made more complicated by the multiple imputation structure. We summarize various predictions constructed from covariates, including multiply imputed covariates, and either the set of imputation-specific prediction model coefficients or the pooled prediction model coefficients. We further describe approaches for using the predictions to assess model performance. We distinguish between ideal model performance and pragmatic model performance, where the former refers to the model's performance in an ideal clinical setting where all individuals have fully observed predictors and the latter refers to the model's performance in a real-world clinical setting where some individuals have missing predictors. The approaches are compared through an extensive simulation study based on the UK700 trial. We determine that measures of ideal model performance can be estimated within imputed datasets and subsequently pooled to give an overall measure of model performance. Alternative methods to evaluate pragmatic model performance are required and we propose constructing predictions either from a second set of covariate imputations which make no use of observed outcomes, or from a set of partial prediction models constructed for each potential observed pattern of covariate. Pragmatic model performance is generally lower than ideal model performance. We focus on model performance within the derivation data, but describe how to extend all the methods to a validation dataset.

Description

Keywords

Measures of model performance, Missing data, Model validation, Multiple imputation, Prediction models, Rubin's rules, Analysis of Variance, Biometry, Clinical Trials as Topic, Humans, Logistic Models, Models, Statistical

Journal Title

Biom J

Conference Name

Journal ISSN

0323-3847
1521-4036

Volume Title

57

Publisher

Wiley-VCH Verlag
Sponsorship
Medical Research Council (MR/L003120/1)
Medical Research Council (G0701619)
Medical Research Council (MR/K014811/1)
British Heart Foundation (None)
Angela Wood part supported by MRC grant G0701619. Ian White from MRC _Biostatistics Unit with unit programme number U105260558