The Trick Doesn’t Work if You’ve Already Seen the Gorilla: How Anticipatory Effects Contaminate Pre-treatment Measures in Field Experiments

Abstract

Objectives: When participants in experiments can anticipate the intervention, the study outcomes are said to be confounded. Ample evidence on intentional and unintentional interferences with the stimuli exists which suggests that participants tend to alter their response to the intervention prior to exposure to it; consequently, the measurement of post-treatment effects has been shown to be contaminated because there is a systematic interaction between measurement and treatment. However, an often-ignored consequence of such anticipatory effects is the impact on the baseline measure. If participants can anticipate the intervention early enough, the pretreatment scores will be conditional, therefore producing a biased estimate of the measure. We explore recent evidence on this bias and present a practical option for the mitigation of anticipatory effects. Methods: A review of the literature across multiple disciplines which addresses concerns regarding anticipatory effects. Results: Pretreatment measures, especially of the dependent variables at their baseline values, can be contaminated by anticipatory effects. We show that the major concerns experimenters should consider in this context are: (1) When can we say that the treatment effect ‘commenced’? (2) What forms the pretesting measure? (3) Are anticipatory effects case-specific or are there industry-wide, global anticipatory effects? (4) What can we conclude from studies whose pretest measures are affected by the anticipated treatment effect? and (5) What solutions are there for anticipatory effects? Conclusions: We outline arguments against the fundamental hypothesis that pre-treatment measurements of baseline measures are unaffected by the study conditions. The implications of anticipatory effects for both research and policy are often ignored, which may lead to erroneous conclusions regarding the treatment’s effectiveness, its benefits being underestimated, or both. The bias can be resolved by collecting ‘clean’ baseline measures prior to the commencement of the anticipatory effects, but the first step is to be aware of their potential.