Neuroinflammation is increasingly recognized as playing a key pathogenetic role in Alzheimer's disease (AD). We examined the relationship between in vivo neuroinflammation and gray matter (GM) changes. Twenty-eight subjects with clinically probable AD (n = 14) and amyloid-positive mild cognitive impairment (n = 14) (age 71.9 ± 8.4 years, 46% female) and 24 healthy controls underwent structural 3T brain MRI. AD/mild cognitive impairment participants exhibited GM atrophy and cortical thinning in AD-related temporoparietal regions (false discovery rate-corrected p < 0.05). Patients also showed increased microglial activation in temporal cortices. Higher 11C-PK11195 binding in these regions was associated with reduced volume and cortical thickness in parietal, occipital, and cingulate areas (false discovery rate p < 0.05). Hippocampal GM atrophy and parahippocampal cortical thinning were related to worse cognition (p < 0.05), but these effects were not mediated by microglial activation. This study demonstrates an association between in vivo microglial activation and markers of GM damage in AD, positioning neuroinflammation as a potential target for immunotherapeutic strategies.