Human perception is remarkably flexible: We experience vivid three-dimensional (3D) structure under diverse conditions, from the seemingly random magic-eye stereograms to the aesthetically beautiful, but obviously flat, canvases of the Old Masters. How does the brain achieve this apparently effortless robustness? Using brain imaging we are beginning to discover how different parts of the visual cortex support 3D perception by tracing different computations in the dorsal and ventral pathways. This review concentrates on studies of binocular disparity and its combination with other depth cues. This work suggests that the dorsal visual cortex is strongly engaged by 3D information and is involved in integrating signals to represent the structure of viewed surfaces. The ventral cortex may store representations of object configurations and the features required for task performance. These differences can be broadly understood in terms of the different computational demands of reducing estimator variance versus increasing the separation between exemplars.