Time series arise all over the sciences and engineering, with numerous important applications in many different fields. In the ‘Big Data Era’, the tasks of time series modelling, inference and prediction have become more critical than ever. In this thesis, we introduce a collection of statistical ideas and algorithmic tools to build a general Bayesian framework for modelling and inference with time series data, which is found to be effective in a number of practical settings, including both discrete and real-valued observations.

For discrete-valued time series, we describe a novel Bayesian framework based on variable-memory Markov chains, called Bayesian Context Trees (BCT). This is a rich class of high-order Markov chains that admit parsimonious representations by allowing the memory of the process to depend on the values of the most recent observations. A general prior structure is introduced and a collection of methodological and algorithmic tools are developed, allowing for efficient, exact Bayesian inference in this setting. It is shown that the evidence (averaging over all models and parameters) can be computed exactly, and that the a posteriori most likely models can be precisely identified. The relevant algorithms have only linear complexity in the length of the data and can be updated sequentially, facilitating online prediction. We provide extensive experimental results illustrating the efficacy of our methods in a number of statistical tasks, as well as theoretical results that further justify their use.

The proposed approach is then extended to real-valued time series, where it is employed to develop a general hierarchical Bayesian framework for building mixture models. At the top level, a set of discrete contexts (or ‘states’) are extracted from quantised versions of the observations. The set of all relevant contexts are represented as a context tree. At the bottom level, a different real-valued time series model is associated with each state. This defines a very general framework that can be used in conjunction with any existing model class to build flexible and interpretable mixture models. We show that, again, effective computational tools can be developed, allowing for efficient, exact Bayesian inference. The utility of the general framework is illustrated when autoregressive models are used as the base model, resulting in a nonlinear AR mixture, and when conditional heteroscedastic models are used, resulting in a flexible mixture model that gives a systematic way of modelling the well-known volatility asymmetries in financial data. The proposed methods are found to outperform the state-of-the-art techniques in various settings, both with simulated and real-world data.