The appearance of quasiparticle excitations with fractional statistics is a remarkable defining trait of topologically ordered systems. In this work, we investigate the experimentally relevant finite temperature regime in which one species of quasiparticle acts as a stochastic background for another, more energetically costly, species that hops coherently across the lattice. The nontrivial statistical angle between the two species leads to interference effects that we study using a combination of numerical and analytical tools. In the limit of self-retracing paths, we are able to use a Bethe lattice approximation to construct exact analytical expressions for the time evolution of the site-resolved density profile of a spinon initially confined to a single site. Our results help us to understand the temperature-dependent crossover from ballistic to quantum (sub-)diffusive behaviour as a consequence of destructive interference between lattice walks. The subdiffusive behaviour is most pronounced in the case of semionic mutual statistics, and it may be ascribed to the localised nature of the effective tight-binding description, an effect that is not captured by the Bethe lattice mapping. In addition to quantum spin liquids, our results are directly applicable to the dynamics of isolated holes in the large-$U$ limit of the Hubbard model, relevant to ultracold atomic experiments. A recent proposal to implement $Z2$ topologically ordered Hamiltonians using quantum annealers provides a further exciting avenue to test our results.