Reinforcement Learning for and with Large Language Models: Architecture, Agency, and Algorithmic Discovery

Abstract

This dissertation establishes a new blueprint for autonomous agency founded on the bidirectional synthesis of reinforcement learning (RL) and large language models (LLMs). We argue that combining RL's decision-making formalism with the LLM's vast world knowledge creates agents that are not only more capable but can also be used to automate the discovery of fundamental learning principles. This work validates this thesis by constructing and solving a deliberate hierarchy of five challenges, progressing from modelling the world to learning how to learn. First, we address the foundational problems of modelling and acting. We introduce RL as a powerful search mechanism for scientific discovery, developing Deep Generative Symbolic Regression (DGSR), a framework that discovers interpretable, symbolic world models by refining a pre-trained generative transformer with policy gradients at inference time. To act within such models, we tackle the long-horizon credit assignment problem in high-frequency continuous control by proposing EvoControl, a bi-level architecture that learns a high-level policy with Proximal Policy Optimization (PPO) and a robust, low-level controller with Evolution Strategies (ES). Next, we unify these components into a single, adaptive agent. We introduce an LLM-based agent architecture that learns and plans online, purely in-context. This agent distils its experiences into a symbolic memory of ``atomic facts'' which are then used to ground a model-based, depth-limited lookahead search, where the LLM itself serves as the latent world model, action proposer, and value function. To overcome the transformer's finite context window, we scale this agent with the L2MAC (Large Language Model Automatic Computer) framework, a von Neumann-inspired architecture featuring a Control Unit, external read/write memory, and a self-correction loop, enabling the execution of arbitrarily complex tasks such as full-stack software generation. Finally, we ascend to a meta-level, using the LLM not just as an agent but as a co-designer in the scientific process. We develop an LLM-driven discovery pipeline that automates the generation and validation of new learning algorithms. This process led to the discovery of DiscoPOP, a novel, non-convex preference optimization objective that outperforms prominent human-designed algorithms. Collectively, this thesis delivers a cohesive set of architectures and algorithms that chart a path from system identification and control to scalable agency and, ultimately, to the automated discovery of the learning rules that will govern future intelligent systems.