The energies of molecular excited states arise as solutions to the electronic Schrödinger equation and are often compared to experiment. At the same time, nuclear quantum motion is known to be important and to induce a redshift of excited state energies. However, it is thus far unclear whether incorporating nuclear quantum motion in molecular excited state calculations leads to a systematic improvement of their predictive accuracy, making further investigation necessary. Here, we present such an investigation by employing two first-principles methods for capturing the effect of quantum fluctuations on excited state energies, which we apply to the Thiel set of organic molecules. We show that accounting for zero-point motion leads to much improved agreement with experiment, compared to "static" calculations that only account for electronic effects, and the magnitude of the redshift can become as large as 1.36 eV. Moreover, we show that the effect of nuclear quantum motion on excited state energies largely depends on the molecular size, with smaller molecules exhibiting larger redshifts. Our methodology also makes it possible to analyze the contribution of individual vibrational normal modes to the redshift of excited state energies, and in several molecules, we identify a limited number of modes dominating this effect. Overall, our study provides a foundation for systematically quantifying the shift of excited state energies due to nuclear quantum motion and for understanding this effect at a microscopic level.