Many animals partition resources to avoid competition, and in colonially-breeding species this often leads to divergent space or habitat use. During the non-breeding season, foraging constraints are relaxed, yet the patterns and drivers of segregation both between and within populations are poorly understood. We modelled habitat preference to examine how extrinsic (habitat availability and intra-specific competition) and intrinsic factors (population, sex and breeding outcome) influence the distributions of non-breeding grey-headed albatrosses $Thalassarche$ chrysostoma tracked from two major populations, South Georgia (Atlantic Ocean) and the Prince Edward Islands (Indian Ocean). Spatial segregation was greater than expected, reflecting distinct seasonal differences in habitat selection and accessibility, and avoidance of intra-specific competition with local breeders. Previously failed birds segregated spatially from successful birds during summer, when they used less productive waters, suggesting a link between breeding outcome and subsequent habitat selection. In contrast, we found weak evidence of sexual segregation, which did not reflect a difference in habitat use. Our results indicate that the large-scale spatial structuring of albatross distributions results from interactions between extrinsic and intrinsic factors, with important implications for population dynamics. As habitat preferences differed substantially between colonies, populations should be considered independently when identifying critical areas for protection.