We exploit the nonlinearity arising from the spin-photon interaction in an InAs quantum dot to demonstrate phase shifts of scattered light pulses at the single-photon level. Photon phase shifts of close to 90$°$ are achieved using a negatively-charged quantum dot in a micropillar cavity. We also demonstrate a photon phase switch by using a spin-pumping mechanism through Raman transitions in an in-plane magnetic field. The experimental findings are supported by a theoretical model which explores the dynamics of the system. Our results demonstrate the potential of quantum dot-induced nonlinearities for quantum information processing.