When divergent populations form hybrids, hybrid fitness can vary with genome composition, current environmental conditions, and the divergence history of the populations. We develop analytical predictions for hybrid fitness, which incorporate all three factors. The predictions are based on Fisher's geometric model, and apply to a wide range of population genetic parameter regimes and divergence conditions, including allopatry and parapatry, local adaptation, and drift. Results show that hybrid fitness can be decomposed into intrinsic effects of admixture and heterozygosity, and extrinsic effects of the (local) adaptedness of the parental lines. Effect sizes are determined by a handful of geometric distances, which have a simple biological interpretation. These distances also reflect the mode and amount of divergence, such that there is convergence toward a characteristic pattern of intrinsic isolation. We next connect our results to the quantitative genetics of line crosses in variable or patchy environments. This means that the geometrical distances can be estimated from cross data, and provides a simple interpretation of the "composite effects." Finally, we develop extensions to the model, involving selectively induced disequilibria, and variable phenotypic dominance. The geometry of fitness landscapes provides a unifying framework for understanding speciation, and wider patterns of hybrid fitness.