Pavement condition monitoring is mainly performed manually. Inspectors are driving or walking the road network bare eyed to look for irregularities. Moreover, processing the collected data for understanding the road condition is also a manual task. In this paper a framework that automates the process is presented. Video data collected from the car’s parking camera is utilized to detect defects in frames. Simultaneously, elevation signals collected from accelerometers attached to the car are processed to reconstruct the profile of the road and detect defects associated with its z-axis, such as bumps. A GPS device is synchronized with the other sensors to acquire the data’s geolocation. Detected defects are then classified according to their type and their severity is assessed. All information is then transferred via 4G network to a central server, where the Road Condition Index of road segments necessary to classify roads is calculated. Finally, everything is saved in a Pavement Management System. Preliminary results on the processing of video data demonstrate the frameworks’ promising application. The initial identification of frames including defects produces an accuracy of 96% and approximately 97% precision. Further experiments on such frames, aiming at the detection of potholes, patches and three different types of cracks result in over 84% overall accuracy and over 85% precision.