The inclusive production rates of isolated, prompt photons in $p$+Pb collisions at $s\mathrm{NN}=8.16$ TeV are studied with the ATLAS detector at the Large Hadron Collider using a dataset with an integrated luminosity of 165 nb$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ recorded in 2016. The cross-section and nuclear modification factor $Rp\mathrm{Pb}$ are measured as a function of photon transverse energy from 20 GeV to 550 GeV and in three nucleon-nucleon centre-of-mass pseudorapidity regions, (-2.83,-2.02), (-1.84,0.91), and (1.09,1.90). The cross-section and $Rp\mathrm{Pb}$ values are compared with the results of a next-to-leading-order perturbative QCD calculation, with and without nuclear parton distribution function modifications, and with expectations based on a model of the energy loss of partons prior to the hard scattering. The data disfavour a large amount of energy loss and provide new constraints on the parton densities in nuclei.