Social cognition, in particular mindreading, enables the understanding of another individual’s feelings, intentions, desires, and mental states. The Reading the Mind in the Eyes Test (RMET) captures the ability to identify mental states from gaze. We investigated RMET accuracy in the context of age and cognition across the whole adult age-range (19–79 years) in a very large population-based sample (N = 1,603) with linear regression models accounting for cognitive abilities, neurological diseases, and psychiatric disorders. Higher age predicted lower RMET performance in women and men, suggesting difficulties to infer mental states from gaze at older age. Effects remained stable when taking other cognitive abilities and psychiatric disorders or neurological diseases into account. Our results show that RMET performance as a measure of social cognition declines with increasing age.