Minor mergers are thought to be responsible for the size growth of quiescent field galaxies with decreasing redshift. We test this hypothesis using the cluster environment as a laboratory. Satellite galaxies in clusters move at high velocities, making mergers between them rare. The stellar mass–size relation in 10 clusters and in the field is measured and compared at z ∼ 1. Our cluster sample contains 344 spectroscopically confirmed cluster members with Gemini/Gemini Multi-Object Spectrographs and 182 confirmed with Hubble Space Telescope/Wide Field Camera 3 G141 grism spectroscopy. On average, quiescent and star-forming cluster galaxies are smaller than their field counterparts by (0.08 ± 0.04) and (0.07 ± 0.01) dex, respectively. These size offsets are consistent with the average sizes of quiescent and star-forming field galaxies between 1.2 ≤ z ≤ 1.5, implying the cluster environment has inhibited size growth between this period and z ∼ 1. The negligible differences measured between the z ∼ 0 field and cluster quiescent mass–size relations in other works imply that the average size of quiescent cluster galaxies must rise with decreasing redshift. Using a toy model, we show that the disappearance of the compact cluster galaxies might be explained if, on average, ∼40 per cent of them merge with their brightest cluster galaxies (BCGs) and ∼60 per cent are tidally destroyed into the intracluster light (ICL) between 0 ≤ z ≤ 1. This is in agreement with the observed stellar mass growth of BCGs between 0 ≤ z ≤ 1 and the observed ICL stellar mass fraction at z ∼ 0. Our results support minor mergers as the cause for the size growth in quiescent field galaxies, with cluster-specific processes responsible for the similarity between the field and cluster quiescent mass–size relations at low redshift.