We investigate constraints on scalar dark matter (DM) by analyzing the Lyman-alpha forest, which probes structure formation at medium and small scales, and also by studying its cosmological consequences at high and low redshift. For scalar DM that constitutes more than 30% of the total DM density, we obtain a lower limit m >~ 10^{-21} eV for the mass of scalar DM. This implies an upper limit on the initial field displacement (or the decay constant for an axion-like field) of phi <~ 10^{16} GeV. We also derive limits on the energy scale of cosmic inflation and establish an upper bound on the tensor-to-scalar ratio of r < 10^{-3} in the presence of scalar DM. Furthermore, we show that there is very little room for ultralight scalar DM to solve the "small-scale crisis" of cold DM without spoiling the Lyman-alpha forest results. The constraints presented in this paper can be used for testing generic theories that contain light scalar fields.