Current Facial Expression Recognition (FER) ap- proaches tend to be insensitive to individual differences in expression and interaction contexts. They are unable to adapt to the dynamics of real-world environments where data is only available incrementally, acquired by the system during interactions. In this paper, we propose a novel continual learning framework with imagination for FER (CLIFER) that (i) implements imagination to simulate expression data for particular subjects and integrates it with (ii) a complementary learning-based dual-memory (episodic and semantic) model, to augment person-specific learning. The framework is evaluated on its ability to remember previously seen classes as well as on generalising to yet unseen classes, resulting in high F1-scores for multiple FER datasets: RAVDESS (episodic: F1= 0.98 ± 0.01, semantic: F1= 0.75 ± 0.01), MMI (episodic: F1= 0.75 ± 0.07, semantic: F1= 0.46 ± 0.04) and BAUM-1 (episodic: F1= 0.87 ± 0.05, semantic: F1= 0.51 ± 0.04).