It is well known that amorphous solids display a phonon spectrum where the Debye $\sim \omega 2$ law at low frequency melds into an anomalous excess-mode peak (the boson peak) before entering a quasi-localized regime at higher frequencies dominated by scattering. The microscopic origin of the boson peak has remained elusive despite various attempts to put it in a clear connection with structural disorder at the atomic/molecular level. Using numerical calculations on model systems, we show that the microscopic origin of the boson peak is directly controlled by the local breaking of center-inversion symmetry. In particular, we find that both the boson peak and the nonaffine softening of the material display a strong positive correlation with a new order parameter describing the local inversion symmetry of the lattice. The standard bond-orientational order parameter, instead, is shown to be a poor correlator and cannot explain the boson peak in randomly-cut crystals with perfect bond-orientational order. Our results bring a unifying understanding of the boson peak anomaly for model glasses and defective crystals in terms of a universal local symmetry-breaking principle of the lattice.