We compute the dielectric response of glasses starting from a microscopic system-bath Hamiltonian of the Zwanzig-Caldeira-Leggett type and using an ansatz from kinetic theory for the memory function in the resulting generalized Langevin equation. The resulting framework requires the knowledge of the vibrational density of states (DOS) as input, which we take from numerical evaluation of a marginally stable harmonic disordered lattice, featuring a strong boson peak (excess of soft modes over Debye $\sim$ ω law). The dielectric function calculated based on this ansatz is compared with experimental data for the paradigmatic case of glycerol at T $\lesssim$ T. Good agreement is found for both the reactive (real) part of the response and for the α-relaxation peak in the imaginary part, with a significant improvement over earlier theoretical approaches. On the low-frequency side of the α peak, the fitting supports the presence of $\sim$ ω modes at vanishing eigenfrequency as recently shown [E. Lerner, G. During, and E. Bouchbinder, Phys. Rev. Lett. 117, 035501 (2016)]. α-wing asymmetry and stretched-exponential behavior are recovered by our framework, which shows that these features are, to a large extent, caused by the soft boson-peak modes in the DOS.