The charge-transport properties of conjugated polymers have been studied extensively for opto-electronic device applications. Some polymer semiconductors not only support the ambipolar transport of electrons and holes, but do so with comparable carrier mobilities. This opens the possibility of gaining deeper insight into the charge-transport physics of these complex materials via comparison between electron and hole dynamics while keeping other factors, such as polymer microstructure, equal. Here, we use field-induced electron spin resonance spectroscopy to compare the spin relaxation behavior of electron and hole polarons in three ambipolar conjugated polymers. Our experiments show unique relaxation regimes as a function of temperature for electrons and holes, whereby at lower temperatures electrons relax slower than holes, but at higher temperatures, in the so-called spin-shuttling regime the trend is reversed. On the basis of theoretical simulations we attribute this to differences in the delocalization of electron and hole wavefunctions and show that spin relaxation in the spin shuttling regimes provides a sensitive probe of the intimate coupling between charge and structural dynamics.