Temperature-dependent surface-enhanced Raman scattering (SERS) is used to investigate the photoluminescence and background continuum always present in SERS but whose origin remains controversial. Both the Stokes and anti-Stokes background is found to be dominated by inelastic light scattering (ILS) from the electrons in the noble metal nanostructures supporting the plasmon modes. The anti-Stokes background is highly temperature dependent and is shown to be related to the thermal occupation of electronic states within the metal via a simple model. This suggests new routes to enhance SERS sensitivities, as well as providing ubiquitous and calibrated real-time temperature measurements of nanostructures.