Proper immune system function hinders cancer development, but little is known about whether genetic variants linked to cancer risk alter immune cells. Here, we report 57 cancer risk loci associated with differences in immune and/or stromal cell contents in the corresponding tissue. Predicted target genes show expression and regulatory associations with immune features. Polygenic risk scores also reveal associations with immune and/or stromal cell contents, and breast cancer scores show consistent results in normal and tumor tissue. SH2B3 links peripheral alterations of several immune cell types to the risk of this malignancy. Pleiotropic SH2B3 variants are associated with breast cancer risk in BRCA1/2 mutation carriers. A retrospective case-cohort study indicates a positive association between blood counts of basophils, leukocytes, and monocytes and age at breast cancer diagnosis. These findings broaden our knowledge of the role of the immune system in cancer and highlight promising prevention strategies for individuals at high risk.