Glasses exhibit a liquidlike structure but a solidlike rheological response with plastic deformations only occurring beyond yielding. Thus, predicting the rheological behavior from the microscopic structure is difficult, but important for materials science. Here, we consider colloidal suspensions and propose to supplement the static structural information with the local dynamics, namely, the rearrangement and breaking of the cage of neighbors. This is quantified by the mean squared nonaffine displacement and the number of particles that remain nearest neighbors for a long time, i.e., long-lived neighbors, respectively. Both quantities are followed under shear using confocal microscopy and are the basis to calculate the affine and nonaffine contributions to the elastic stress, which is complemented by the viscoelastic stress to give the total stress. During start-up of shear, the model predicts three transient regimes that result from the interplay of affine, nonaffine, and viscoelastic contributions. Our prediction quantitatively agrees with rheological data and their dependencies on volume fraction and shear rate.