Physical activity has positive effects on brain health and cognitive function throughout the life span. Thus far, few studies have examined the effects of physical activity on white matter microstructure and psychomotor speed within the same, population-based sample (critical if conclusions are to extend to the wider population). Here, using diffusion tensor imaging and a simple reaction time task within a relatively large population-derived sample (N = 399; 18-87 years) from the Cambridge Centre for Ageing and Neuroscience (Cam-CAN), we demonstrate that physical activity mediates the effect of age on white matter integrity, measured with fractional anisotropy. Higher self-reported daily physical activity was associated with greater preservation of white matter in several frontal tracts, including the genu of corpus callosum, uncinate fasciculus, external capsule, and anterior limb of the internal capsule. We also show that the age-related slowing is mediated by white matter integrity in the genu. Our findings contribute to a growing body of work, suggesting that a physically active lifestyle may protect against age-related structural disconnection and slowing.