Quantum mechanics imposes a limit on the precision of a continuous position measurement of a harmonic oscillator, due to backaction arising from quantum fluctuations in the measurement field. This standard quantum limit can be surpassed by monitoring only one of the two non-commuting quadratures of the motion, known as backaction-evading measurement. This technique has not been implemented using optical interferometers to date. Here we demonstrate, in a cavity optomechanical system operating in the optical domain, a continuous two-tone backaction-evading measurement of a localized gigahertz-frequency mechanical mode of a photonic-crystal nanobeam cryogenically and optomechanically cooled close to the ground state. Employing quantum-limited optical heterodyne detection, we explicitly show the transition from conventional to backaction-evading measurement. We observe up to 0.67 dB (14%) reduction of total measurement noise, thereby demonstrating the viability of backaction-evading measurements in nanomechanical resonators for optical ultrasensitive measurements of motion and force.