We consider the motion of a classical colored spinless particle under the influence of an external Yang-Mills potential $A$ on a compact manifold with boundary of dimension $\ge 3$. We show that under suitable convexity assumptions, we can recover the potential $A$, up to gauge transformations, from the lens data of the system, namely, scattering data plus travel times between boundary points.