Deforestation has been widespread in the Northern Albertine Rift Landscape in rural Western Uganda. In this paper, we present perceptions from local residents and narratives from key informants on causes of forest loss during a 30-year period between 1985 and 2014. While the generic drivers we find are consistent with previous literature, we suggest that the specific context in which forest cover is lost in rural areas is path dependent, and this is vital for adequate management. In the Ugandan case, the history of the sugar industry and its relation to local political drivers and international considerations (e.g. biofuel) are prominent. Global drivers of forest loss therefore mask local-level complexities, but an amalgamation of local-level dynamics does not necessarily sum up to larger-scale manifestations (in a linear manner): striking a balance between understanding local-level and large-scale dynamics could be key in addressing the deforestation conundrum. We surveyed 263 households in 7 parishes around Budongo and Bugoma forests, and conducted 22 key informant interviews. Our findings indicate that the drivers and mechanisms of deforestation are local; they also vary between Budongo and Bugoma. Key amongst these include: agricultural expansion (28%–58.5% of the responses)—with large-scale commercial and small-scale subsistence farming significant around Budongo and Bugoma respectively; “poverty” (26%–76%) often alluding to heavy dependence on forests for livelihoods. Others include: population growth driven by dissimilar migratory patterns; and moving protected forest boundaries. Our data suggest that a combination of both local and key informant perceptions is instrumental in filling data gaps where a dearth of information is prevalent (especially around Bugoma forest), and is important for corroboration of other scientific data (e.g. remote sensing). However, a survey of wider literature indicates that there are significant issues missing from their stated views. While the continued expansion of cash-crop farming and lack of inclusion of local people in forest policy continues to raise concern, the stability of protected forest (i.e. Budongo and Bugoma) is encouraging and suggests a refocusing of the forest debate on practical working schemes for forest preservation and recovery might be the way forward for sustainable forestry and livelihoods.