State-specific complete active space self-consistent field (SS-CASSCF) theory has emerged as a promising route to accurately predict electronically excited energy surfaces away from molecular equilibria. However, its accuracy and practicality for chemical systems of photochemical interest have yet to be fully determined. We investigate the performance of the SS-CASSCF theory for the low-lying ground and excited states in the double bond rotation of ethylene. We show that state-specific approximations with a minimal (2e,2o) active space provide comparable accuracy to state-averaged calculations with much larger active spaces, while optimizing the orbitals for each excited state significantly improves the spatial diffusivity of the wave function. However, the incorrect ordering of state-specific solutions causes excited state solutions to coalesce and disappear, creating unphysical discontinuities in the potential energy surface. Our findings highlight the theoretical challenges that must be overcome to realize practical applications of state-specific electronic structure theory for computational photochemistry.