Ly$\alpha$ forest data probing the post-reionization Universe show surprisingly large opacity fluctuations over rather large (≥50 $h-1$ comoving Mpc) spatial scales. We model these fluctuations using a hybrid approach utilizing the large-volume Millennium simulation to predict the spatial distribution of QSOs combined with smaller scale full hydrodynamical simulation performed with $RAMSES$ and post-processed with the radiative transfer code $ATON$. We produce realistic mock absorption spectra that account for the contribution of galaxies and QSOs to the ionizing UV background. These improved models confirm our earlier findings that a significant ($\gtrsim$50 per cent) contribution of ionizing photons from QSOs can explain the large reported opacity fluctuations on large scales. The inferred QSO luminosity function is thereby consistent with recent estimates of the space density of QSOs at this redshift. Our simulations still somewhat struggle, however, to reproduce the very long (110 $h-1$ comoving Mpc) high-opacity absorption through observed in ULAS J0148+0600, perhaps suggesting an even later end of reionization than assumed in our previously favoured model. Medium-deep/medium area QSO surveys as well as targeted searches for the predicted strong transverse QSO proximity effect would illuminate the origin of the observed large-scale opacity fluctuations. They would allow us to substantiate whether UV fluctuations due to QSO are indeed primarily responsible, or whether significant contributions from other recently proposed mechanisms such as large-scale fluctuations in temperature and mean free path (even in the absence of rare bright sources) are required.