We use deep Chandra X-ray imaging to measure the distribution of specific black hole accretion rates ($LX$ relative to the stellar mass of the galaxy) and thus trace AGN activity within star-forming and quiescent galaxies, as a function of stellar mass (from $108.5-1011.5M\odot$) and redshift (to $z\sim 4$). We adopt near-infrared selected samples of galaxies from the CANDELS and UltraVISTA surveys, extract X-ray data for every galaxy, and use a flexible Bayesian method to combine these data and to measure the probability distribution function of specific black hole accretion rates, $\lambda sBHAR$. We identify a broad distribution of $\lambda sBHAR$ in both star-forming and quiescent galaxies---likely reflecting the stochastic nature of AGN fuelling---with a roughly power-law shape that rises toward lower $\lambda sBHAR$, a steep cutoff at $\lambda sBHAR\gtrsim 0.1-1$ (in Eddington equivalent units), and a turnover or flattening at $\lambda sBHAR\lesssim 10-3-10-2$. We find that the probability of a star-forming galaxy hosting a moderate $\lambda sBHAR$ AGN depends on stellar mass and evolves with redshift, shifting toward higher $\lambda sBHAR$ at higher redshifts. This evolution is truncated at a point corresponding to the Eddington limit, indicating black holes may self-regulate their growth at high redshifts when copious gas is available. The probability of a quiescent galaxy hosting an AGN is generally lower than that of a star-forming galaxy, shows signs of suppression at the highest stellar masses, and evolves strongly with redshift. The AGN duty cycle in high-redshift ($z\gtrsim 2$) quiescent galaxies thus reaches $\sim$20 per cent, comparable to the duty cycle in star-forming galaxies of equivalent stellar mass and redshift.