Infrared excesses around metal-polluted white dwarfs have been associated with the accretion of dusty planetary material. This work analyses the available infrared data for an unbiased sample of white dwarfs and demonstrates that no more than 3.3 per cent can have a wide, flat, opaque dust disc, extending to the Roche radius, with a temperature at the disc inner edge of Tin = 1400 K, the standard model for the observed excesses. This is in stark contrast to the incidence of pollution of about 30 per cent. We present four potential reasons for the absence of an infrared excess in polluted white dwarfs, depending on their stellar properties and inferred accretion rates: (i) their dust discs are opaque, but narrow, thus evading detection if more than 85 per cent of polluted white dwarfs have dust discs narrower than δr < 0.04r, (ii) their dust discs have been fully consumed, which only works for the oldest white dwarfs with sinking time-scales longer than hundreds of years, (iii) their dust is optically thin, which can supply low accretion rates of <10$\mathrm{<mrow\; data-mjx-texclass="ORD">7</mrow>}$ gs$\mathrm{<mrow\; data-mjx-texclass="ORD">-1</mrow>}$ if dominated by (Poynting–Robertson) PR-drag, and higher accretion rates, if inwards transport of material is enhanced, e.g. due to the presence of gas, (iv) their accretion is supplied by a pure gas disc, which could result from the sublimation of optically thin dust for T$\mathrm{<mrow\; data-mjx-texclass="ORD">\ast </mrow>}$ > 20 000 K. Future observations sensitive to faint infrared excesses or the presence of gas can test the scenarios presented here, thereby better constraining the nature of the material fuelling accretion in polluted white dwarfs.