Monitoring of construction workers is important in managing labour productivity. To date, the construction sector relies either on intensive manual observations or intrusive tag-based practices. Visual tracking methods can provide automated and tag-less monitoring. However, no method to date has succeeded in tracking multiple workers, as construction sites are complex environments due to congestion, background clutter and occlusions. In addition, workers have similar appearance and exhibit illumination/scale/posture variations and abrupt changes in movement over the course of their task. To address these shortcomings, we propose a vision-based method that consists of 3 models. Firstly, an adaptive model provides continuous information about the previous position of workers and their appearance features. Secondly, a prediction model is used to calculate the current position of workers, and finally, an appearance model provides accurate localisation. Experimental results show that the proposed method achieves high performance and outperforms the latest relevant state of the art methods.