Construction is an inherently less safe sector than other sectors because it exposes workers to harsh and dangerous working environments. The nature of the construction industry results in a comparatively high incidence of serious injuries and death caused by falls from a height, musculoskeletal disorders and being struck by objects. This paper presents a new concept that can tackle this problem in the future. The central hypothesis of this study is that it is possible to eliminate injuries if we move the human construction worker off-site and remotely link his/her motions to a Robotic Construction Worker (RCW) on-site. As a first steppingstone towards this ultimate goal, two systems essential for the RCW were developed in this study. First, a novel system that combines 3D body and hand position tracking was developed to capture the movements of human construction worker. This combination of tracking enables the capture of changes in the orientations and articulations of the entire human body. Second, a real-time simulation system that connects a human construction worker off-site to a virtual RCW was developed to demonstrate the proposed concept in a variety of construction scenarios. The simulation results demonstrate the future viability of the RCW concept and indicate the promise of this system for eliminating the health and safety risks faced by human construction workers.