jats:pUser authentication is key in user authorization on smart and personal devices. Over the years, several authentication mechanisms have been proposed: these also include behavioral-based biometrics. However, behavioral-based biometrics suffer from two issues: they are prone to degradation in performance (accuracy) over time (e.g., due to data distribution changes arising from user behavior) and the need to learn the machine learning model from scratch, when adding new users. In this paper, we propose ContAuth, a system that can enhance the robustness of behavioral-based authentication. ContAuth continuously adapts to new incoming data (data incremental learning) and is able to add new users without retraining (class incremental learning). Specifically, ContAuth combines deep learning models with online learning models to achieve learning on the fly, thereby preventing a severe drop in the accuracy between sessions (over time). To add new users, ContAuth employs class incremental learning methods. We evaluate ContAuth on multiple behavior-based user authentication modalities: breathing, gait. and EMG. Our results show that our framework can help True Positive Rate (TPR) to remain high (>85 %) compared to other methods for all the modalities except EMG (>70%) across the sessions while keeping False Positive Rates (FPR) at a minimum (0-10%). It can achieve up to 35% improvement in TPR over a traditional deep learning model. Additionally, iCaRL (an incremental learning method) enables ContAuth to allow the addition of new users by alleviating catastrophic forgetting, to a large extent. Finally, we also show that ContAuth can be deployed efficiently and effectively on device, further providing data privacy.</jats:p>