A detailed temporal analysis of the spectral diffusion phenomenon in single photon emitting InGaN/GaN quantum dots (QDs) is performed via measurements of both time-varying emission spectra and single photon emission intensity autocorrelation times. Excitation dependent phenomena are investigated via the optical excitation of carriers into the GaN barrier material and also directly into the InGaN. Excitation into the InGaN reveals that the fastest environmental fluctuations occur on timescales as long as a few hundreds of nanoseconds: an order of magnitude longer than previously measured in GaN QDs. Such long time scales may in future allow for the generation of indistinguishable photons in spite of the fact that the experimentally measured linewidths are broad.