Much of the recent success of machine learning methods was enabled by exploiting the wealth of labeled data produced in the past few years. However, for several important real-world applications such large-scale data collection is still infeasible. This includes areas such as robotics, healthcare, Earth science, and chemistry, where acquiring data can be expensive and time-consuming.

In this thesis, we consider three different learning problems where the amount of data that can be collected is limited. This includes settings with restricted access to labels, entire datasets, and generated experience during online learning. We address these data limitations by adopting sequential decision-making strategies, which iterate between collecting new data and making informed decisions based on newly acquired evidence.

First, we tackle the problem of how to efficiently collect batches of labels when the cost of acquiring labels is high. Probabilistic active learning methods can be used to greedily select the most informative data points to be labeled. However, for many large-scale problems, standard greedy procedures become computationally infeasible. To mitigate this issue, we introduce a scalable Bayesian batch active learning approach that is motivated by approximating the complete data posterior of the model parameters.

Second, we address the challenge of automating the design of molecules to accelerate the search for novel drugs and materials. Since only a small region of the chemical space has been explored so far, the amount of data available for certain chemical systems is limited. We overcome the dataset-dependence of generative models for 3D molecular design by formulating this problem as a reinforcement learning task, and propose a symmetry-aware policy that can generate molecular structures unattainable with previous methods.

Lastly, we consider the problem of how to efficiently learn robot behaviors across different tasks. A promising direction towards this goal is to generalize locally learned policies over different task contexts. Contextual policy search offers data-efficient learning and generalization by explicitly conditioning the policy on a parametric context space. We further structure the contextual policy representation, leading to faster learning and better generalization in various robotic domains.