Testosterone supplementation is commonly used for its effects on sexual function, bone health and body composition, yet its effects on disease outcomes are unknown. To better understand this, we identified genetic determinants of testosterone levels and related sex hormone traits in 425,097 UK Biobank study participants. Using 2,571 genome-wide significant associations, we demonstrate that the genetic determinants of testosterone levels are substantially different between sexes and that genetically higher testosterone is harmful for metabolic diseases in women but beneficial in men. For example, a genetically determined 1 s.d. higher testosterone increases the risks of type 2 diabetes (odds ratio (OR) = 1.37 (95% confidence interval (95% CI): 1.22-1.53)) and polycystic ovary syndrome (OR = 1.51 (95% CI: 1.33-1.72)) in women, but reduces type 2 diabetes risk in men (OR = 0.86 (95% CI: 0.76-0.98)). We also show adverse effects of higher testosterone on breast and endometrial cancers in women and prostate cancer in men. Our findings provide insights into the disease impacts of testosterone and highlight the importance of sex-specific genetic analyses.