Robust Pothole Detection through Dual-pass RGB and Depth Fusion

Abstract

Having potholes detected and repaired is essential for road maintenance and uninterrupted transport. Generations of researchers and engineers have been innovating on using computer vision and sensors to detect potholes, however, gaps in knowledge remain in the lack of robustness of detectors trained with RGB images in varied environments and the inability to harness depth in detecting potholes with monocular images. This research proposes a solution by fusing predictions made by RGB-trained detectors and monocular depth estimation. The solution first orthorectifies the pavement from perspective RGB images. It then receives predictions from RGB-trained detectors and a depth detector enabled by DINOv2 independently. These predictions are subsequently fused by weighted bounding box fusion and have masks predicted by Segment Anything. Whereas RGB-trained detectors perform well in test sets within the training context, they show a drastic loss of performance in out-of-context situations, such as in images taken on different roads in more challenging environmental conditions. Fusing predictions with depth enhances F1 scores by 16% to 81% in out-of-context situations, reinforcing detection robustness. This solution paves the way for further research on expanding detection to motorways and overcoming shadows in images.