The Antarctic Peninsula experienced a rapid rise in regional temperature during the second half of the 20th century, but the regional pattern of multi-centennial temperature changes and their dynamical drivers remain poorly understood. Here we use proxies of biological productivity in rare, deep moss banks to infer past surface air temperature changes on the Antarctic Peninsula and identify the drivers of these changes. Late Holocene temperatures are broadly consistent between the low-elevation moss bank records and a high-elevation ice core site, and we conclude that variation in the strength of the westerlies, linked to the Southern Annular Mode, is the most likely driver. Our data do not support a hypothesized persistent temperature dipole over the Antarctic Peninsula related to a strong influence of El Niño–Southern Oscillation. Rates of change in biological productivity on the peninsula over the 20th century are unusual in the context of the late Holocene, and further warming will drive rapid future increases in moss growth and microbial populations.