Earth's surface topography is a direct physical expression of our planet's dynamics. Most is isostatic, controlled by variations in thickness and density within the crust and lithosphere, but a significant proportion arises due to forces exerted by underlying mantle convection. This dynamic topography directly connects the evolution of surface environments to Earth's deep interior, but it remains poorly understood: predictions from mantle flow simulations are often inconsistent with inferences from the geological record, with little consensus about its spatial pattern, wavelength and amplitude. Here, we demonstrate that previous comparisons between predictive models and observational constraints have been biased by subjective choices. Using measurements of residual topography beneath the world's oceans, and an innovative statistical approach to performing spherical harmonic analyses, we generate a robust estimate of Earth's oceanic residual topography power spectrum. Our analyses imply power of 0.5 +- 0.35 km^2 and peak amplitudes of 0.8 +- 0.1 km at long-wavelength (~10^4 km), decreasing by roughly one order of magnitude at shorter wavelengths (~10^3 km). We show that geodynamical simulations can only be reconciled with these observational constraints if they incorporate lithospheric structure and its impact on global mantle flow, illustrating that both deep (long-) and shallow (shorter-wavelength) processes are crucial to generating the observed surface response. Our results imply that dynamic topography is intimately connected to the structure and evolution of Earth's lithosphere, presenting a challenge to the reconstruction of its temporal evolution and impact at Earth's surface.