Recent distance ladder determinations of the Hubble constant $H0$ disagree at about the $3.5\sigma$ level with the value determined from Planck measurements of the cosmic microwave background (CMB) assuming a $\Lambda$CDM cosmology. This discrepancy has prompted speculation that new physics might be required beyond that assumed in the $\Lambda$CDM model. In this paper, we apply the inverse distance ladder to fit a parametric form of $H(z)$ to baryon acoustic oscillation (BAO) and Type Ia supernova data together with priors on the sound horizon at the end of the radiation drag epoch, $rd$. We apply priors on $rd$, based on inferences from either Planck or the Wilkinson Microwave Anistropy Probe (WMAP), and demonstrate that these values are consistent with CMB-independent determinations of $rd$ derived from measurements of the primordial deuterium abundance, BAO and supernova data assuming the $\Lambda$CDM cosmology. The $H(z)$ constraints that we derive are independent of detailed physics within the dark sector at low redshifts, relying only on the validity of the Friedmann-Robertson-Walker (FRW) metric of General Relativity. For each assumed prior on $rd$, we find consistency with the inferred value of $H0$ and the Planck $\Lambda$CDM value and corresponding tension with the distance ladder estimate.