Identifying the underlying genetic drivers of the heritability of breast cancer prognosis remains elusive. We adapt a network-based approach to handle underpowered complex datasets to provide new insights into the potential function of germline variants in breast cancer prognosis. This network-based analysis studies ~7.3 million variants in 84,457 breast cancer patients in relation to breast cancer survival and confirms the results on 12,381 independent patients. Aggregating the prognostic effects of genetic variants across multiple genes, we identify four gene modules associated with survival in estrogen receptor (ER)-negative and one in ER-positive disease. The modules show biological enrichment for cancer-related processes such as G-alpha signaling, circadian clock, angiogenesis, and Rho-GTPases in apoptosis.