The transforming growth factor beta (TGF-β) signaling pathway exerts opposing effects on cancer cells, acting as either a tumor promoter or a tumor suppressor. Here, we show that these opposing effects are a result of the synergy between SMAD3, a downstream effector of TGF-β signaling, and the distinct epigenomes of breast-tumor-initiating cells (BTICs). These effects of TGF-β are associated with distinct gene expression programs, but genomic SMAD3 binding patterns are highly similar in the BTIC-promoting and BTIC-suppressing contexts. Our data show cell-type-specific patterns of DNA and histone modifications provide a modulatory layer by determining accessibility of genes to regulation by TGF-β/SMAD3. LBH, one such context-specific target gene, is regulated according to its DNA methylation status and is crucial for TGF-β-dependent promotion of BTICs. Overall, these results reveal that the epigenome plays a central and previously overlooked role in shaping the context-specific effects of TGF-β in cancer.