We present an active feedback scheme acting continuously on the state of a quantum gas dispersively coupled to a high-finesse optical cavity. The quantum gas is subject to a transverse pump laser field inducing a self-organization phase transition, where the gas acquires a density modulation and photons are scattered into the resonator. Photons leaking from the cavity allow for a real-time and non-destructive readout of the system. We stabilize the mean intra-cavity photon number through a micro-processor controlled feedback architecture acting on the intensity of the transverse pump field. The feedback scheme can keep the mean intra-cavity photon number $nph$ constant, in a range between $nph=0.17\pm 0.04$ and $nph=27.6\pm 0.5$, and for up to 4 s. Thus we can engage the stabilization in a regime where the system is very close to criticality as well as deep in the self-organized phase. The presented scheme allows us to approach the self-organization phase transition in a highly controlled manner and is a first step on the path towards the realization of many-body phases driven by tailored feedback mechanisms.