When a spatially uniform temperature change is imposed on a solid with more than one phase, or on a polycrystal of a single, non-cubic phase (showing anisotropic expansion-contraction), the resulting thermal strain is inhomogeneous (non-affine). Thermal cycling induces internal stresses, leading to structural and property changes that are usually deleterious. Glasses are the solids that form on cooling a liquid if crystallization is avoided--they might be considered the ultimate, uniform solids, without the microstructural features and defects associated with polycrystals. Here we explore the effects of cryogenic thermal cycling on glasses, specifically metallic glasses. We show that, contrary to the null effect expected from uniformity, thermal cycling induces rejuvenation, reaching less relaxed states of higher energy. We interpret these findings in the context that the dynamics in liquids become heterogeneous on cooling towards the glass transition, and that there may be consequent heterogeneities in the resulting glasses. For example, the vibrational dynamics of glassy silica at long wavelengths are those of an elastic continuum, but at wavelengths less than approximately three nanometres the vibrational dynamics are similar to those of a polycrystal with anisotropic grains. Thermal cycling of metallic glasses is easily applied, and gives improvements in compressive plasticity. The fact that such effects can be achieved is attributed to intrinsic non-uniformity of the glass structure, giving a non-uniform coefficient of thermal expansion. While metallic glasses may be particularly suitable for thermal cycling, the non-affine nature of strains in glasses in general deserves further study, whether they are induced by applied stresses or by temperature change.