Magmatism in subduction zones builds continental crust and causes most of Earth's subaerial volcanism. The production rate and composition of magmas are controlled by the thermal structure of subduction zones. A range of geochemical and heat flow evidence has recently converged to indicate that subduction zones are hotter at lithospheric depths beneath the arc than predicted by canonical thermomechanical models, which neglect magmatism. We show that this discrepancy can be resolved by consideration of the heat transported by magma. In our one- and two-dimensional numerical models and scaling analysis, magmatic transport of sensible and latent heat locally alters the thermal structure of canonical models by $\sim$300 K, increasing predicted surface heat flow and mid-lithospheric temperatures to observed values. We find the advection of sensible heat to be larger than the deposition of latent heat. Based on these results we conclude that thermal transport by magma migration affects the chemistry and the location of arc volcanoes.