Quantifying the genetic correlation between cancers can provide important insights into the mechanisms driving cancer etiology. Using genome-wide association study summary statistics across six cancer types based on a total of 296,215 cases and 301,319 controls of European ancestry, here we estimate the pair-wise genetic correlations between breast, colorectal, head/neck, lung, ovary and prostate cancer, and between cancers and 38 other diseases. We observed statistically significant genetic correlations between lung and head/neck cancer (rg = 0.57, p = 4.6 × 10-8), breast and ovarian cancer (rg = 0.24, p = 7 × 10-5), breast and lung cancer (rg = 0.18, p =1.5 × 10-6) and breast and colorectal cancer (rg = 0.15, p = 1.1 × 10-4). We also found that multiple cancers are genetically correlated with non-cancer traits including smoking, psychiatric diseases and metabolic characteristics. Functional enrichment analysis revealed a significant excess contribution of conserved and regulatory regions to cancer heritability. Our comprehensive analysis of cross-cancer heritability suggests that solid tumors arising across tissues share in part a common germline genetic basis.