Structured illumination can reject out-of-focus signal from a sample, enabling high-speed and high-contrast imaging over large areas with widefield detection optics. However, this optical sectioning technique is currently limited by image reconstruction artefacts and poor performance at low signal-to-noise ratios. We combine multicolour interferometric pattern generation with machine learning to achieve high-contrast, real-time reconstruction of image data that is robust to background noise and sample motion. We validate the method in silico and demonstrate imaging of diverse specimens, from fixed and live biological samples to synthetic biosystems, reconstructing data live at 11 Hz across a 44 × 44μm2 field of view, and demonstrate image acquisition speeds exceeding 154 Hz.