Cdc42-GTP is required for apical domain formation in epithelial cells, where it recruits and activates the Par-6-aPKC polarity complex, but how the activity of Cdc42 itself is restricted apically is unclear. We used sequence analysis and 3D structural modeling to determine which Drosophila GTPase-activating proteins (GAPs) are likely to interact with Cdc42 and identified RhoGAP19D as the only high-probability Cdc42GAP required for polarity in the follicular epithelium. RhoGAP19D is recruited by α-catenin to lateral E-cadherin adhesion complexes, resulting in exclusion of active Cdc42 from the lateral domain. rhogap19d mutants therefore lead to lateral Cdc42 activity, which expands the apical domain through increased Par-6/aPKC activity and stimulates lateral contractility through the myosin light chain kinase, Genghis khan (MRCK). This causes buckling of the epithelium and invasion into the adjacent tissue, a phenotype resembling that of precancerous breast lesions. Thus, RhoGAP19D couples lateral cadherin adhesion to the apical localization of active Cdc42, thereby suppressing epithelial invasion.