In microbial communities, each species often has multiple, distinct phenotypes, but studies of ecological stability have largely ignored this subpopulation structure. Here, we show that such implicit averaging over phenotypes leads to incorrect linear stability results. We then analyze the effect of phenotypic switching in detail in an asymptotic limit and partly overturn classical stability paradigms: abundant phenotypic variation is linearly destabilizing but, surprisingly, a rare phenotype such as bacterial persisters has a stabilizing effect. Finally, we extend these results by showing how phenotypic variation modifies the stability of the system to large perturbations such as antibiotic treatments.