Halide perovskites have shown promise to advance the field of light detection in next generation photodetectors, offering performance and functionality beyond what is currently possible with traditional inorganic semiconductors. Despite a relatively high density of defects in perovskite thin films, long carrier diffusion lengths and lifetimes suggest that many defects are benign. However, perovskite photodetectors show detection behaviour that varies with time, creating inconsistent device performance and difficulties in accurate characterisation. Here, we link the changing behaviour to mobile defects that migrate through perovskites, leading to detector currents that drift on the timescale of seconds. These effects not only complicate reproducible device performance, but also introduces characterisation challenges. We demonstrate that such transient phenomena generate measurement artefacts that mean the value of specific detectivity measured can vary by up to two orders of magnitude even in the same device. The presence of defects can lead to photoconductive gain in photodetectors, and we show batch-to-batch processing variations in perovskite devices gives varying degrees of charge carrier injection and photocurrent amplification under low light intensities. We utilise the passivating effect of ageing to reduce the impact of defects, minimising current drifts and eliminating the gain. This work highlights the potential issues arising from mobile defects, which lead to inconsistent photodetector operation, and identifies the potential for defects to tune photodetection behaviour in perovskite photodetectors.