Improving our understanding of user trial samples using survey data

Abstract

User research such as user trials provides valuable information on users and how they respond to interfaces in practice. However, it can be hard to ensure a repre-sentative sample. We propose a methodology to improve the understanding of a sample’s skew and to identify the characteristics of those who are missing by com-paring the sample with survey data. This can improve the interpretation of results and inform further recruitment to improve the sample. We provide a case study of this methodology in practice. 30 participants were recruited using quota sampling with significant effort to obtain people with low technology experience. Neverthe-less, comparison with UK survey data on technology experience, competence and attitudes identified four key groups of people not included in the sample, covering 29% of the population. We discuss how these missing people would likely respond on the tasks, based on the characteristics of similar people in the survey