Abstract: Background: Observational studies have shown that milk consumption is inversely associated with colorectal, bladder, and breast cancer risk, but positively associated with prostate cancer. However, whether the associations reflect causality remains debatable. We investigated the potential causal associations of milk consumption with the risk of colorectal, bladder, breast, and prostate cancer using a genetic variant near the LCT gene as proxy for milk consumption. Methods: We obtained genetic association estimates for cancer from the UK Biobank (n = 367,643 women and men), FinnGen consortium (n = 135,638 women and men), Breast Cancer Association Consortium (n = 228,951 women), and Prostate Cancer Association Group to Investigate Cancer Associated Alterations in the Genome consortium (n = 140,254 men). Milk consumption was proxied by a genetic variant (rs4988235 or rs182549) upstream of the gene encoding lactase, which catalyzes the breakdown of lactose. Results: Genetically proxied milk consumption was associated with a reduced risk of colorectal cancer. The odds ratio (OR) for each additional milk intake increasing allele was 0.95 (95% confidence interval [CI] 0.91–0.99; P = 0.009). There was no overall association of genetically predicted milk consumption with bladder (OR 0.99; 95% CI 0.94–1.05; P = 0.836), breast (OR 1.01; 95% CI 1.00–1.02; P = 0.113), and prostate cancer (OR 1.01; 95% CI 0.99–1.02; P = 0.389), but a positive association with prostate cancer was observed in the FinnGen consortium (OR 1.07; 95% CI 1.01–1.13; P = 0.026). Conclusions: Our findings strengthen the evidence for a protective role of milk consumption on colorectal cancer risk. There was no or limited evidence that milk consumption affects the risk of bladder, breast, and prostate cancer.