A novel search for exotic decays of the Higgs boson into pairs of long-lived neutral particles, each decaying into a bottom quark pair, is performed using 139 fb$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ of $s=13$ TeV proton-proton collision data collected with the ATLAS detector at the LHC. Events consistent with the production of a Higgs boson in association with a leptonically decaying $Z$ boson are analysed. Long-lived particle (LLP) decays are reconstructed from inner-detector tracks as displaced vertices with high mass and track multiplicity relative to Standard Model processes. The analysis selection requires the presence of at least two displaced vertices, effectively suppressing Standard Model backgrounds. The residual background contribution is estimated using a data-driven technique. No excess over Standard Model predictions is observed, and upper limits are set on the branching ratio of the Higgs boson to LLPs. Branching ratios above 10% are excluded at 95% confidence level for LLP mean proper lifetimes $c\tau$ as small as 4 mm and as large as 100 mm. For LLP masses below 40 GeV, these results represent the most stringent constraint in this lifetime regime.