The functions of the tumor microenvironment (TME) are orchestrated by precise spatial organization of specialized cells, yet little is known about the multicellular structures that form within the TME. Here we systematically mapped TME structures in situ using imaging mass cytometry and multitiered spatial analysis of 693 breast tumors linked to genomic and clinical data. We identified ten recurrent TME structures that varied by vascular content, stromal quiescence versus activation, and leukocyte composition. These TME structures had distinct enrichment patterns among breast cancer subtypes, and some were associated with genomic profiles indicative of immune escape. Regulatory and dysfunctional T cells co-occurred in large 'suppressed expansion' structures. These structures were characterized by high cellular diversity, proliferating cells and enrichment for BRCA1 and CASP8 mutations and predicted poor outcome in estrogen-receptor-positive disease. The multicellular structures revealed here link conserved spatial organization to local TME function and could improve patient stratification.