The space of inflationary models is vast, containing wide varieties of mechanisms, symmetries, and spectra of particles. Consequently, the space of observational signatures is similarly complex. Hence, it is natural to look for boundaries of the space of models and their signatures. In this paper, we explore the possible symmetries associated with the primordial cosmological perturbations and their correlators in the asymptotic future. Assuming the observed homogeneity, isotropy and (approximate) scale invariance, we prove three main results. First, correlation functions of scalar metric fluctuations are uniquely characterized by soft theorems and are free from ambiguity under field redefinitions. Second, whatever the particle content and interactions, when the standard soft theorems apply, invariance under de Sitter boosts (linearly realized conformal invariance) is only possible if all connected correlators vanish identically, i.e. if the theory is free. Third, conformal invariance is the largest set of linearly realized (bosonic) symmetries of the correlators of any single scalar, irrespectively of any soft theorems or particle content.