Exact expressions are derived for the intermediate scattering function (ISF) of a quantum particle diffusing in a harmonic potential and linearly coupled to a harmonic bath. The results are valid for arbitrary strength and spectral density of the coupling. The general, exact non-Markovian result is expressed in terms of the classical velocity autocorrelation function, which represents an accumulated phase during a scattering event. The imaginary part of the exponent of the ISF is proportional to the accumulated phase, which is an antisymmetric function of the correlation time $t$. The expressions extend previous results given in the quantum Langevin framework where the classical response of the bath was taken as Markovian. For a special case of non-Markovian friction, where the friction kernel decays exponentially in time rather than instantaneously, we provide exact results relating to unconfined quantum diffusion, and identify general features that allow insight to be exported to more complex examples. The accumulated phase as a function of the t has a universal gradient at the origin, depending only on the mass of the diffusing system particle. At large t the accumulated phase reaches a constant limit that depends only on the classical diffusion coefficient and is therefore independent of the detailed memory properties of the friction kernel. Non-Markovian properties of the friction kernel are encoded in the details of how the accumulated phase switches from its $t\to -\infty$ to its $t\to -\infty$ limit, subject to the constraint of the universal gradient. When memory effects are significant, the transition from one limit to the other becomes non-monotonic, owing to oscillations in the classical velocity autocorrelation. The result is interpreted in terms of a solvent caging effect, in which slowly fluctuating bath modes create transient wells for the system particle.