We study the effect of different feedback prescriptions on the properties of the low redshift ($z\le 1.6$) Ly$\alpha$ forest using a selection of hydrodynamical simulations drawn from the Sherwood simulation suite. The simulations incorporate stellar feedback, AGN feedback and a simplified scheme for efficiently modelling the low column density Ly$\alpha$ forest. We confirm a discrepancy remains between Cosmic Origins Spectrograph (COS) observations of the Ly$\alpha$ forest column density distribution function (CDDF) at $z\simeq 0.1$ for high column density systems ($NHI>1014cm-2$), as well as Ly$\alpha$ velocity widths that are too narrow compared to the COS data. Stellar or AGN feedback -- as currently implemented in our simulations -- have only a small effect on the CDDF and velocity width distribution. We conclude that resolving the discrepancy between the COS data and simulations requires an increase in the temperature of overdense gas with $\Delta =4$--$40$, either through additional He$,\rm \scriptstyle II\ $ photo-heating at $z>2$ or fine-tuned feedback that ejects overdense gas into the IGM at just the right temperature for it to still contribute significantly to the Ly$\alpha$ forest. Alternatively a larger, currently unresolved turbulent component to the line width could resolve the discrepancy.