Brain age is a widely used index for quantifying individuals' brain health as deviation from a normative brain aging trajectory. Higher-than-expected brain age is thought partially to reflect above-average rate of brain aging. Here, we explicitly tested this assumption in two independent large test datasets (UK Biobank [main] and Lifebrain [replication]; longitudinal observations ≈ 2750 and 4200) by assessing the relationship between cross-sectional and longitudinal estimates of brain age. Brain age models were estimated in two different training datasets (n ≈ 38,000 [main] and 1800 individuals [replication]) based on brain structural features. The results showed no association between cross-sectional brain age and the rate of brain change measured longitudinally. Rather, brain age in adulthood was associated with the congenital factors of birth weight and polygenic scores of brain age, assumed to reflect a constant, lifelong influence on brain structure from early life. The results call for nuanced interpretations of cross-sectional indices of the aging brain and question their validity as markers of ongoing within-person changes of the aging brain. Longitudinal imaging data should be preferred whenever the goal is to understand individual change trajectories of brain and cognition in aging.