Ridge functions have recently emerged as a powerful set of ideas for subspace-based dimension reduction. In this paper we begin by drawing parallels between ridge subspaces, sufficient dimension reduction and active subspaces, contrasting between techniques rooted in statistical regression and those rooted in approximation theory. This sets the stage for our new algorithm that approximates what we call a Gaussian ridge function---the posterior mean of a Gaussian process on a dimension-reducing subspace---suitable for both regression and approximation problems. To compute this subspace we develop an iterative algorithm that alternates between optimizing over the Stiefel manifold to compute the subspace and optimizing the hyperparameters of the Gaussian process. We demonstrate the utility of the algorithm on two analytical functions, where we obtain near exact ridge recovery, and a turbomachinery case study, where we compare the efficacy of our approach with three well-known sufficient dimension reduction methods: SIR, SAVE, and CR. The comparisons motivate the use of the posterior variance as a heuristic for identifying the suitability of a dimension-reducing subspace.