While quantum mechanics imposes a fundamental limit on the precision of interferometric measurements of mechanical motion due to measurement backaction, the nonlinear nature of the coupling also leads to parametric instabilities that place practical limits on the sensitivity by limiting the power in the interferometer. Such instabilities have been extensively studied in the context of gravitational wave detectors, and their presence has recently been reported in Advanced LIGO. Here, we observe experimentally and describe theoretically a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements, designed to overcome the standard quantum limit, and demonstrate the effect in the optical domain with a photonic crystal nanobeam, and in the microwave domain with a micromechanical oscillator coupled to a microwave resonator. In contrast to the well-known oscillatory parametric instability that occurs in single-tone, blue-detuned pumping, which is characterized by a vanishing effective mechanical damping, the parametric instability in balanced two-tone optomechanics is exponential, and is a result of small detuning errors in the two pump frequencies. Its origin can be understood in a rotating frame as the vanishing of the effective mechanical frequency due to an optical spring effect. Counterintuitively, the instability occurs even in the presence of perfectly balanced intracavity fields, and can occur for both signs of detuning. We find excellent quantitative agreement with our theoretical predictions. Since the constraints on tuning accuracy become stricter with increasing probe power, it imposes a fundamental limitation on BAE measurements, as well as other two-tone schemes. In addition to introducing a new limitation in two-tone BAE measurements, the results also introduce a new type of nonlinear dynamics in cavity optomechanics.