A Survey of Reinforcement Learning For Economics

Abstract: This survey (re)introduces reinforcement learning methods to economists. The curse of dimensionality limits how far exact dynamic programming can be effectively applied, forcing us to rely on suitably "small" problems or our ability to convert "big" problems into smaller ones. While this reduction has been sufficient for many classical applications, a growing class of economic models resists such reduction. Reinforcement learning algorithms offer a natural, sample-based extension of dynamic programming, extending tractability to problems with high-dimensional states, continuous actions, and strategic interactions. I review the theory connecting classical planning to modern learning algorithms and demonstrate their mechanics through simulated examples in pricing, inventory control, strategic games, and preference elicitation. I also examine the practical vulnerabilities of these algorithms, noting their brittleness, sample inefficiency, sensitivity to hyperparameters, and the absence of global convergence guarantees outside of tabular settings. The successes of reinforcement learning remain strictly bounded by these constraints, as well as a reliance on accurate simulators. When guided by economic structure, reinforcement learning provides a remarkably flexible framework. It stands as an imperfect, but promising, addition to the computational economist's toolkit. A companion survey (Rust and Rawat, 2026b) covers the inverse problem of inferring preferences from observed behavior.

• The paper rigorously unifies reinforcement learning and dynamic programming, detailing classic algorithms like Q-learning, policy gradients, and actor-critic methods.
• The survey presents empirical findings that reveal both rapid policy convergence under controlled conditions and significant limitations of model-free methods in high-dimensional settings.
• It examines practical applications in economic control, structural estimation, and dynamic pricing, highlighting challenges in off-policy evaluation and causal inference.

A Comprehensive Survey of Reinforcement Learning for Economic Applications

Introduction and Motivation

The paper "A Survey of Reinforcement Learning For Economics" (2603.08956) provides an extensive synthesis of the intersection between reinforcement learning (RL) and economics. It is specifically tailored for economists seeking to understand RL's foundational algorithms, their theoretical underpinnings, and their applicability to economic environments. The paper systematically unifies RL with classical dynamic programming (DP) and explores RL's value— and limitations— as a tool for high-dimensional economic models, dynamic control, strategic games, bandit settings, causal inference, and preference learning.

Unifying RL and Dynamic Programming: Theory and Algorithmic Foundations

A key contribution of the survey is its rigorous unification of RL and DP. Both frameworks solve Markov decision processes using the Bellman equation but differ operationally: DP requires explicit transition/reward modeling and enables broad, batch updates; RL incrementally updates value estimates using observed transitions, allowing for scalable, sample-based learning.

The survey traces the theoretical lineage of RL from animal psychology and early game-playing programs to Bellman's formalization of DP. It then presents a detailed technical analysis of canonical RL algorithms and their relationships:

• Monte Carlo, Temporal Difference (TD), and Q-learning: Monte Carlo reduces variance by averaging returns but is limited to episodic tasks; TD learning introduces bootstrapping to enable incremental updating. Q-learning solves for the optimal action-value function without an explicit model, making it central to model-free RL.
• Policy Gradient and Actor-Critic Methods: Policy gradient methods (REINFORCE, natural policy gradient) directly optimize parametric policies, addressing continuous action spaces. Actor-critic algorithms combine value-function estimation (critic) with policy updates (actor) on separate timescales.
• Deep RL and Modern Algorithms: Extensions to high-dimensional or continuous domains are covered, including Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), Trust Region Policy Optimization (TRPO), and Soft Actor-Critic (SAC), each addressing key issues such as instability and sample inefficiency in prior RL approaches.

A central theoretical advance emphasized is the interpretation of policy iteration as Newton's method applied to the Bellman equation, which explains superlinear convergence in structured environments.

Empirical Insights into RL Performance and Convergence

Empirical benchmarks using classic economic (the Brock-Mirman growth model) and stylized reinforcement learning environments illustrate algorithmic properties and critical weaknesses:

Figure 1: The Brock–Mirman economy demonstrates contraction rates and Newton-like convergence behavior of value and policy iteration, respectively.

The paper shows that, while planning algorithms converge rapidly with access to models, model-free RL is statistically and computationally limited by exploration noise and function approximation. The experimental analysis in deterministic gridworlds and classical economics benchmarks found:

• Q-learning/Q($\lambda$) converge reliably to optimal policies if sufficient exploration is ensured.
• On-policy methods (SARSA, REINFORCE, NPG, PPO) achieve high expected returns but often fail to recover the value function at off-equilibrium states, underscoring a critical limitation for counterfactual evaluation and policy transfer.

  Figure 2: Learned value functions at convergence checkpoints illustrate that only certain RL algorithms recover $V^*$ globally.

  

  Figure 3: Visualization of policy convergence for multiple algorithms, revealing persistent errors in off-policy or poorly explored regions.

These findings articulate that empirical success in RL does not always translate to full recovery of the optimal policy or value function, a caveat of significant interest in economic applications involving counterfactual reasoning.

RL in Applied Economic Control and Structural Modeling

The survey catalogs a broad spectrum of RL applications in economics, highlighting practical deployments in ride-hailing (DiDi, Lyft), data center optimization (Google), revenue management (hotels, finance), and inventory control. It also conducts a simulation study of RL-based fleet replacement, showing scaling behavior in classical dynamic programming versus DQN-based RL:

Figure 4: Computation time and discounted returns for dynamic programming, DQN, and heuristics in a multi-engine fleet replacement problem.

Notably, while RL methods match dynamic programming benchmarks in tractable settings, they become essential as the state space grows combinatorially and DP becomes infeasible, especially in multi-agent or high-dimensional economics environments.

Structural Estimation, Dynamic Discrete Choice, and RL Algorithms

The paper summarizes how RL techniques—most notably temporal-difference (TD) learning, Q-learning, and policy gradient—enable structural estimation for dynamic discrete choice models:

• TD-based estimators for conditional choice probability (CCP) and value function computation circumvent the need for transition density estimation, facilitating structural analysis with high-dimensional state variables.
• Policy gradient approaches allow estimation in models with unobserved state variables or complex endogenous decision processes.

These innovations have enabled the estimation of discrete choice models and dynamic games at scales previously considered infeasible, with theoretical guarantees (e.g., $\sqrt{n}$-consistency for semi-gradient estimators) when combined with locally robust inference.

RL in Games, Dynamic Oligopoly, and Mechanism Design

The survey reviews advances in using RL for equilibrium computation in dynamic economic games and auctions, as well as multi-agent learning:

• Counterfactual Regret Minimization (CFR) and Neural Extensions: CFR and CFR+ scale to large extensive-form and imperfect-information games, matching known theoretical predictions for Nash equilibrium convergence rates $O(1/\sqrt{T})$ and, in practice, $O(1/T)$ with CFR+.
• Dynamic Oligopoly and Strategic Learning: RL provides tractable means to compute experience-based equilibria and dynamic best responses in oligopoly, auctions, and innovation-driven industries, with empirical studies highlighting potential reversals of static conclusions (e.g., the welfare impact of mergers under dynamic innovation incentives).

Bandit Models and Real-Time RL in Economics

A substantial section is devoted to bandit-based online learning settings, such as dynamic pricing and assortment optimization. The survey synthesizes the regret landscape, explicitly quantifying how added economic structure, revealed preference axioms, and knowledge of error distributions lower the minimax regret scaling from $O(\sqrt{T})$ to $O(\log T)$ or $O(d\log T)$ in high-dimensional contextual settings. It explicitly demonstrates the pitfalls of ignoring confounding, strategic buyer response, or model misspecification.

Figure 5: Cumulative regret for various dynamic pricing strategies, showing the scaling advantage conferred by economic structure and preference-based eliminations.

Reinforcement Learning from Human Feedback (RLHF)

The survey includes an advanced treatment of RLHF, detailing both the standard preference-based reward modeling pipeline (Bradley-Terry/logit-inferred rewards, PPO finetuning) and direct preference optimization (DPO), which bypasses explicit reward estimation using variational arguments.

Empirical results are presented for gridworld environments where neural reward models and structural representations are compared for sample efficiency in preference-based policy learning:

Figure 6: Stochastic gridworld for RLHF sample complexity studies, with the optimal policy navigating obstacles and traps.

Figure 7: Policy return as a function of human preference comparisons for RLHF with neural and structural reward models.

The results demonstrate nontrivial sample complexity for robust RLHF, and highlight significant challenges in reward misspecification and online versus offline comparison collection.

Offline RL, Causal Inference, and Identification

An advanced portion of the survey deals with offline RL and causal inference in economics, formalizing the confounded MDP and demonstrating the necessity of backdoor-adjusted estimators for off-policy evaluation under endogenous or confounded behavior policies.

• The paper provides explicit identification results using the backdoor criterion and explores extensions via the front-door criterion, instrumental variables, and proximal causal inference, discussing their economic analogs.
• A simulation study quantifies bias amplification in naive estimators and verifies the effectiveness of the backdoor adjustment.

  Figure 8: Bias of off-policy evaluation estimators as a function of confounding strength, highlighting the necessity of proper causal adjustment.

Implications, Limitations, and Future Directions

The survey carefully qualifies the promise and limits of RL in economics:

• Brittleness and Sample Inefficiency: RL algorithms remain algorithmically brittle, sensitive to hyperparameterization, and vulnerable to the deadly triad (off-policy learning, bootstrapping, function approximation), particularly outside tabular or linear settings.
• Model-based vs Model-free: When models can be specified accurately, classic DP or model-based RL is optimal; RL's advantage is greatest where models are unavailable, incomplete, or massively high-dimensional.
• Role of Economic Structure: Incorporating economic constraints (revenue management primitives, WARP, rational inattention) can sharply improve sample efficiency and convergence rates in RL.
• Causal and Empirical Rigor Needed: Causal identification, robust inference for preference learning, and theoretical guarantees under function approximation remain critical research areas.

Practically, RL is positioned as a flexible but imperfect addition to the computational economist’s toolkit, suitable for high-dimensional simulation, policy design, and empirical structural estimation, but not yet a panacea for robust economic modeling.

Conclusion

This survey provides a technically robust, integrative synthesis of reinforcement learning as applied to economics, connecting theoretical foundations, empirical properties, and practical deployment. It demonstrates that RL extends but does not supplant dynamic programming, and that progress in RL for economics will require continued cross-disciplinary work at the intersection of computational mathematics, microeconomic theory, and empirical identification. The survey will serve as a crucial reference for economists and RL researchers seeking to understand the methodological frontiers and outstanding challenges in deploying RL within high-stakes economic environments.