This is a continuation of our previous work with Botvinnik on the nontriviality of the secondary index invariant on spaces of metrics of positive scalar curvature, in which we take the fundamental group of the manifolds into account. We show that the secondary index invariant associated to the vanishing of the Rosenberg index can be highly nontrivial, for positive scalar curvature Spin manifolds with torsionfree fundamental groups which satisfy the Baum--Connes conjecture. For example, we produce a compact Spin 6-manifold such that its space of positive scalar curvature metrics has each rational homotopy group infinite dimensional. At a more technical level, we introduce the notion of "stable metrics" and prove a basic existence theorem for them, which generalises the Gromov--Lawson surgery technique, and we also give a method for rounding corners of manifold with positive scalar curvature metrics.