Masonry arch bridges have proven to be durable and underpin much of the world’s transport infrastructure. However, they are an aging asset, and their effective management is challenging. Existing management focuses heavily on manual inspection, which has been shown as subjective and often leads to incomplete records regarding damage and retrofit of the structure. Improved data collection and processing technologies now provide the opportunity to create a digital record of the entire bridge surface. However, if this digital record is still inspected manually, subjectivity will persist. While considerable progress has been made in automated image-based defect detection of concrete and asphalt infrastructure, relatively little progress has been made for masonry. In this context, a new framework, or pipeline, for automated inspection of masonry arch bridges is presented: from the data capture phase all the way to the diagnosis of the underlying problems with the bridge. After presenting this framework, the focus of this work is on the automated detection of defects within realistic, deteriorated masonry surfaces that are typical of historic masonry arch bridges. This has involved the creation of a pixel-wise annotated dataset of masonry arch bridge surfaces for both different defect classes and mortar joints. This dataset is believed to be unparalleled in both scope and scale, compared to other works in the literature and therefore serves as an invaluable tool for future research in this area. Methodologies for mapping the mortar joints on masonry surfaces have been examined, including a comparison of a hardcoded deterministic algorithm based on pattern detection, with a semantic deep learning model. Both methods performed well; the deep learning model was more robust to noisier image conditions. The effect of mortar joints on automated defect detection procedures was then investigated, as these joints are one of the key distractors and differentiators between masonry and concrete/asphalt surfaces. Whilst simple models based on edge detection were unable to differentiate defects from the masonry surface without prior segmentation of mortar, a more robust classifier based on a Convolutional Neural Network architecture performed well. In addition, prior segmentation of mortar in the image is shown to have no impact on the classifier performance. Use of Class Activation Mapping shows that in most cases the classifier has learnt to ignore the mortar joint interface. The developed classifier performed equal to or better than current manual inspection procedures.