The azimuthal anisotropy of charged particles produced in $s\mathrm{NN}=8.16$ TeV $p$+Pb collisions is measured with the ATLAS detector at the LHC. The data correspond to an integrated luminosity of $165$ $\mathrm{nb}-1$ that was collected in 2016. Azimuthal anisotropy coefficients, elliptic $v2$ and triangular $v3$, extracted using two-particle correlations with a non-flow template fit procedure, are presented as a function of particle transverse momentum ($pT$) between $0.5$ and $50$ GeV. The $v2$ results are also reported as a function of centrality in three different particle $pT$ intervals. The results are reported from minimum-bias events and jet-triggered events, where two jet $pT$ thresholds are used. The anisotropies for particles with $pT$ less than about $2$ GeV are consistent with hydrodynamic flow expectations, while the significant non-zero anisotropies for $pT$ in the range $9$-$50$ GeV are not explained within current theoretical frameworks. In the $pT$ range $2$-$9$ GeV, the anisotropies are larger in minimum-bias than in jet-triggered events. Possible origins of these effects, such as the changing admixture of particles from hard scattering and the underlying event, are discussed.