Working memory (WM) skills are closely associated with learning progress in key areas such as reading and mathematics across childhood. As yet, however, little is known about how the brain systems underpinning WM develop over this critical developmental period. The current study investigated whether and how structural brain correlates of components of the working memory system change over development. Verbal and visuospatial short-term and working memory were assessed in 153 children between 5.58 and 15.92 years, and latent components of the working memory system were derived. Fractional anisotropy and cortical thickness maps were derived from T1-weighted and diffusion-weighted MRI and processed using eigenanatomy decomposition. There was a greater involvement of the corpus callosum and posterior temporal white matter in younger children for performance associated with the executive part of the working memory system. For older children, this was more closely linked with the thickness of the occipitotemporal cortex. These findings suggest that increasing specialization leads to shifts in the contribution of neural substrates over childhood, moving from an early dependence on a distributed system supported by long-range connections to later reliance on specialized local circuitry. Our findings demonstrate that despite the component factor structure being stable across childhood, the underlying brain systems supporting working memory change. Taking the age of the child into account, and not just their overall score, is likely to be critical for understanding the nature of the limitations on their working memory capacity.