Let N be a smooth manifold that is homeomorphic but not diffeomorphic to a closed hyperbolic manifold M. In this paper, we study the extent to which N admits as much symmetry as M. Our main results are examples of N that exhibit two extremes of behavior. On the one hand, we find N with maximal symmetry, i.e. Isom(M) acts on N by isometries with respect to some negatively curved metric on N. For these examples, Isom(M) can be made arbitrarily large. On the other hand, we find N with little symmetry, i.e. no subgroup of Isom(M) of "small" index acts by diffeomorphisms of N. The construction of these examples incorporates a variety of techniques including smoothing theory and the Belolipetsky-Lubotzky method for constructing hyperbolic manifolds with a prescribed isometry group.