The influence of the microstructure on the visco-elastic properties of concrete is investigated using finite element simulations at the meso-scale. We first derive a constitutive law for the creep of the cement paste which accounts for both recoverable and permanent deformations, as well as the influence of temperature and internal relative humidity. The model is calibrated on a set of experiments at the cement paste scale, and then validated after upscaling to the concrete scale. The model is then applied to study the influence of the microstructure on the macroscopic creep of concrete. We show that materials with finer particles exhibit less creep, and that the anisotropy of creep can be explained with the shape and orientation of the aggregates. Furthermore, the acceleration of the stress relaxation in the presence of damage is explained by the micro-mechanical interaction between the aggregates, the cement paste, and the micro-cracks.