Negative symptoms occur in several major mental health disorders with undetermined mechanisms and unsatisfactory treatments; identification of their neural correlates might unveil the underlying pathophysiological basis and pinpoint the therapeutic targets. In this study, participants with major depressive disorder (n = 24), schizophrenia (n = 22), and healthy controls (n = 20) were assessed with 10 frequently used negative symptom scales followed by principal component analysis (PCA) of the scores. A linear model with the prominent components identified by PCA was then regressed on gray and white-matter volumes estimated from T1-weighted magnetic resonance imaging. In depressed patients, negative symptoms such as blunted affect, alogia, withdrawal, and cognitive impairment, assessed mostly via clinician-rated scales were inversely associated with gray matter volume in the bilateral cerebellum. In patients with schizophrenia, anhedonia, and avolition evaluated via self-rated scales inversely related to white-matter volume in the left anterior limb of internal capsule/anterior thalamic radiation and positively in the left superior longitudinal fasiculus. The pathophysiological mechanisms underlying negative symptoms might differ between depression and schizophrenia. These results also point to future negative symptom scale development primarily focused on detecting and monitoring the corresponding changes to brain structure or function.