Repository logo
 

Robust in-vehicle heartbeat detection using multimodal signal fusion

Published version
Peer-reviewed

Repository DOI


Change log

Authors

Warnecke, Joana M. 
Lasenby, Joan 
Deserno, Thomas M. 

Abstract

A medical check-up during driving enables the early detection of diseases. Heartbeat irregularities indicate possible cardiovascular diseases, which can be determined with continuous health monitoring. Therefore, we develop a redundant sensor system based on electrocardiography (ECG) and photoplethysmography (PPG) sensors attached to the steering wheel, a red, green, and blue (RGB) camera behind the steering wheel. For the video, we integrate the face recognition engine SeetaFace to detect landmarks of face segments continuously. Based on the green channel, we derive colour changes and, subsequently, the heartbeat. We record the ECG, PPG, video, and reference ECG with body electrodes of 19 volunteers during different driving scenarios, each lasting 15 min: city, highway, and countryside. We combine early, signal-based late, and sensor-based late fusion with a hybrid convolutional neural network (CNN) and integrated majority voting to deliver the final heartbeats that we compare to the reference ECG. Based on the measured and the reference heartbeat positions, the usable time was 51.75%, 58.62%, and 55.96% for the driving scenarios city, highway, and countryside, respectively, with the hybrid algorithm and combination of ECG and PPG. In conclusion, the findings suggest that approximately half the driving time can be utilised for in-vehicle heartbeat monitoring.

Description

Acknowledgements: Our sincere thanks go out to all the volunteers who contributed to the data acquisition process.


Funder: Technische Universität Braunschweig (1042)

Keywords

Journal Title

Scientific Reports

Conference Name

Journal ISSN

2045-2322

Volume Title

13

Publisher

Nature Publishing Group UK
Sponsorship
Lower Saxony “Vorab” of the Volkswagen Foundation and Center for Digital Innovations (ZN3491)