The common theme of this thesis is the concept of using Monte Carlo techniques to approximate a sequence of probability distributions. Novel methodological contributions are found in Chapter 3 through to Chapter 6. In Chapter 3 we derive a method for the complete characterisation of online statistical models where Monte Carlo approximations are defined sequentially as new data arrive. We then demonstrate the utility of this method in Chapter 4 for the compelling application of de-noising sequential GPS coordinates to be restricted to a road network. In Chapter 5 and Chapter 6, the sequence of probability distributions are defined artificially in order to gradually (and more effectively) approach a single offline target probability distribution. Chapter 5 adopts ideas from high-dimensional time series to efficiently tackle the difficult setting where we cannot evaluate the density of the target distribution and instead can only generate synthetic data. Chapter 6 explores the use of a scalable Hessian approximation in the more common scenario where the target density can be evaluated and even differentiated. Finally, Chapter 7 describes a general purpose software package that can be used to implement and customise all of the discussed algorithms at competitive speeds.