Understanding the origin of strong galactic outflows and the suppression of star formation in dwarf galaxies is a key problem in galaxy formation. Using a set of radiation-hydrodynamic simulations of an isolated dwarf galaxy embedded in a $1010M\odot$ halo, we show that the momentum transferred from resonantly scattered Lyman-$\alpha$ (Lya) photons is an important source of stellar feedback which can shape the evolution of galaxies. We find that Lya feedback suppresses star formation by a factor of two in metal-poor galaxies by regulating the dynamics of star-forming clouds before the onset of supernova explosions (SNe). This is possible because each Lya photon resonantly scatters and imparts 10-300 times greater momentum than in the single scattering limit. Consequently, the number of star clusters predicted in the simulations is reduced by a factor of $\sim 5$, compared to the model without the early feedback. More importantly, we find that galactic outflows become weaker in the presence of strong Lya radiation feedback, as star formation and associated SNe become less bursty. We also examine a model in which radiation field is arbitrarily enhanced by a factor of up to 10, and reach the same conclusion. The typical mass loading factors in our metal-poor dwarf system are estimated to be $\sim 5-10$ near the mid plane, while it is reduced to $\sim 1$ at larger radii. Finally, we find that the escape of ionizing radiation and hence the reionization history of the Universe is unlikely to be strongly affected by Lya feedback.