Language Models as Forecasters

Abstract

Forecasting is a central challenge in both human decision-making and artificial intelligence. Unlike tasks grounded in fixed or verifiable knowledge, forecasting concerns unfolding future events whose outcomes are not yet determined and therefore must be reasoned about under uncertainty. From geopolitical and economic developments to policy analysis and institutional planning, many real-world decisions depend on anticipating macroscopic events rather than confirming what is already known. As such, forecasting discrete, real world events provides a fundamental test of an AI system’s ability to represent the world, estimate likelihoods, and adapt its beliefs as new evidence emerges. This thesis investigates language models as forecasters, explicitly treating them as predictors of real-world events that resolve in natural language or bounded scalar formats, rather than as numerical time-series extrapolators. It examines how they assess what is plausible, how accurately they predict these specific macroscopic outcomes, how well they calibrate their confidence, and whether they revise their beliefs when new information becomes available. Forecasting in this bounded semantic domain serves as a unifying task that links several levels of reasoning under uncertainty, from judging the plausibility of unseen events to generating and updating predictions about the unfolding future. The thesis makes three main contributions. First, it introduces PRobELM, a bench mark that evaluates whether language models can rank plausible over implausible macroscopic scenarios using their internal knowledge, establishing qualitative plausibility reasoning as a precursor to forecasting. Second, it presents FOReCAst, a framework for evaluating semantic predictions and confidence calibration across Boolean, temporal, and quantitative forecasting questions tied to real-world outcomes. Third, it proposes EvolveCast, a dynamic forecasting benchmark that measures how models revise their discrete predictions when given new textual evidence after training. Together, these contributions provide a coherent progression from evaluating plausibility to static event forecasting to dynamic belief updating, supported by new datasets, evaluation protocols, and OSS. The findings show that while language models display emerging forecasting abilities in these domains, they remain limited in three key areas: distinguishing causal plausibility from surface familiarity, maintaining calibrated uncertainty across diverse real-world question types, and updating beliefs with appropriate sensitivity to new narrative information.