The past 40 years in Southeast Asia have seen about 50% of lowland rainforests converted to oil palm and other plantations, and much of the remaining forest heavily logged. Little is known about how fragmentation influences recovery and whether climate change will hamper restoration. Here, we use repeat airborne LiDAR surveys spanning the hot and dry 2015-16 El Niño Southern Oscillation event to measure canopy height growth across 3,300 ha of regenerating tropical forests spanning a logging intensity gradient in Malaysian Borneo. We show that the drought led to increased leaf shedding and branch fall. Short forest, regenerating after heavy logging, continued to grow despite higher evaporative demand, except when it was located close to oil palm plantations. Edge effects from the plantations extended over 300 metres into the forests. Forest growth on hilltops and slopes was particularly impacted by the combination of fragmentation and drought, but even riparian forests located within 40 m of oil palm plantations lost canopy height during the drought. Our results suggest that small patches of logged forest within plantation landscapes will be slow to recover, particularly as ENSO events are becoming more frequent.