Base Policy Prediction

• Base policy prediction is a formal framework that models baseline policies with probabilistic generative models and accounts for systematic suboptimality in decision-making.
• It leverages Bayesian inference and scalable algorithms like MFVI and sequence models to update predictions in real-time and perform off-policy evaluations.
• The approach is applied across human behavior modeling, multi-agent systems, and robotics, demonstrating significant improvements in prediction accuracy and control performance.

Base policy prediction is a formal methodology for inferring, modelling, or evaluating a policy that serves as a baseline in decision-making systems, human behavioral models, multi-agent systems, or control applications. It encompasses both the theoretical foundations for representing uncertainty and suboptimality in policy inference (e.g., as distributions over policies), and scalable algorithmic frameworks for prediction, comparison, and adaptation in the presence of systematic biases, confounding, or unmodelled disturbances. The concept finds applications ranging from systematic suboptimality modelling in human behavior (Laidlaw et al., 2022), off-policy evaluation and uncertainty quantification in contextual bandits and multi-agent systems (Kuipers et al., 2024, Taufiq et al., 2022), to the design of locomotion controllers for legged mobile manipulators (Ma et al., 2022).

1. Mathematical Formulations of Base Policy Distributions

Base policy prediction frequently entails defining a probabilistic generative model over policies. For example, the Boltzmann Policy Distribution (BPD) models human decision-making as draws from a distribution over policies, rather than simply noisy trajectories. Formally, given an MDP $\mathcal{M} = (S, A, P, R, \gamma)$, a stochastic policy $\pi: S \to \Delta(A)$ is assigned the prior

$$p(\pi) = \prod_{s\in S} \mathrm{Dir}(\pi(\cdot|s); \alpha)$$

with concentration parameter $\alpha$, and is "tilted" by exponentiated expected return via

$$P_{\mathrm{BPD}}(\pi) \propto \exp(\beta J(\pi)) p(\pi)$$

where $$J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[\sum_{t=0}^\infty \gamma^t R(s_t,a_t) \right]$$ and $\beta$ controls rationality (Laidlaw et al., 2022).

In settings where the base policy (e.g., status quo human policy or behavioral policy in bandit logging) is unknown or only partially specified, off-policy methods approximate the outcome or regret of switching to a new target policy using the logged data and causal identification assumptions (instrumental variable, marginal sensitivity, proxy variable bounds) (Guerdan et al., 2024, Taufiq et al., 2022, Kuipers et al., 2024). In multi-agent systems, the behavioral policy is formalized as a joint density over trajectories $\beta(\tau) = P^{\pi^b}(\tau)$, with target policies inducing a distributional shift.

2. Bayesian Inference and Online Adaptation

Bayesian updating is key to adapting base policy predictions given new observations. In the BPD framework, after observing a sequence $(s_0,a_0), \ldots, (s_{t-1},a_{t-1})$, the posterior over policies is

$$P(\pi \mid \text{history}) \propto P_{\mathrm{BPD}}(\pi) \prod_{i=0}^{t-1} \pi(a_i|s_i)$$

and the predictive distribution for future actions is the expectation under this posterior (Laidlaw et al., 2022).

Two scalable inference methods are provided:

• Mean-field variational inference (MFVI) maintains a Gaussian factorized posterior over latent variables.
• Sequence models (e.g., Transformers/LSTMs) are trained to directly predict actions conditioned on history, effectively marginalizing the latent space empirically (Laidlaw et al., 2022).

Off-policy evaluation in contextual bandits and multi-agent systems similarly uses importance-weighted conformal prediction: calibration scores are reweighted by the ratio of target-to-behavioral policy, and prediction regions (intervals, sets, joint regions) are constructed to guarantee coverage under the target process (Kuipers et al., 2024, Taufiq et al., 2022).

3. Capturing Systematic Suboptimality and Mutual Information

Standard Boltzmann-rational models, which assign noisy but optimal action selection per-state (maximum-entropy policy), cannot capture systematic suboptimality where behavioral deviations are consistent across states and time. The BPD explicitly couples all actions over an episode by drawing a single policy for the latent sequence, yielding non-zero conditional mutual information between temporally separated actions $$\mathbb{I}(a_t; a_{t^\prime} \mid s_t, s_{t^\prime})$$. This allows for rapid adaptation when a systematic bias is observed early in a trajectory, unlike the trajectory-level maximum-entropy approach with zero mutual information in action sequences (Laidlaw et al., 2022).

In multi-agent off-policy prediction, systematic policy shifts induce distributional changes in the joint state-action trajectory space; methods such as MA-COPP (multi-agent conformal off-policy prediction) exploit closed-form density ratios for ego agents switching policies, thereby characterizing the uncertainty due to base policy change and avoiding exhaustive output space enumeration (Kuipers et al., 2024).

4. Algorithmic Procedures and Implementation Details

Approximate inference over policy distributions or regret bounds relies on generative models, adversarial training, and sample-efficient calibration:

• BPD prior is approximated using latent-variable policy networks ($z\sim N(0,I_d)$, $\pi_z(a|s) = f_\theta(s, z)$) trained via policy gradient (PPO), with adversarial entropy regularization from a discriminator $D_\phi(\pi)$ (Laidlaw et al., 2022).
• Off-policy regret bounds use cross-fitted plugin estimators and doubly robust bias correction, achieving $n^{-1/2}$ rates even under slow nuisance function convergence (Guerdan et al., 2024).

Base policy prediction in robotics applications involves training offline with randomized disturbance sequences, then substituting real model-predictive-control (MPC)-generated wrench forecasts at deployment, enabling disturbance-aware locomotion policies independent of the manipulator's identity (Ma et al., 2022).

Key hyperparameters, architecture choices, and training procedures include:

• Latent dimension $d\approx 1000$ for BPD prior expressiveness.
• Adam optimizer (lr=$1e^{-3}$, momentum $\beta_1 = 0.5$) for GAN-like training loops.
• Conformal calibration sample sizes $N$, simulation-based suffix generation ($M$ samples) for MA-COPP scalability.

5. Empirical Insights and Performance Comparisons

Empirical results indicate that base policy prediction frameworks can achieve predictive and collaborative performance competitive with high-data imitation learners:

• In Overcooked-style environments, BPD (MFVI and Transformer) models reach 0.95-1.12 bits action prediction cross-entropy and 46-53% next-action accuracy, matching behavior cloning with $1/10$ to $1/100$ the human data (Laidlaw et al., 2022).
• Human-AI collaboration mean returns with BPD+PPO reach 50-70, close to BC best-response but requiring no human trajectory pre-collection.
• MA-COPP achieves nominal prediction region coverage (95%) under agent policy shift in high-dimensional multi-agent contexts, with manageable region size and computational tractability ($O(M \cdot d)$) (Kuipers et al., 2024).
• In disturbance-aware locomotion, providing 0.8s of future wrench prediction yields a 208% improvement in tolerable lateral push and precision adaptation to unseen manipulators (Ma et al., 2022).
• In resource allocation (algorithmic decision making), the marginal value of expanding access often dominates improving prediction when resources are scarce, governed by sharp bounds on the Prediction–Access Ratio (PAR) (Perdomo, 2023).

6. Design Principles and Trade-offs

In resource-constrained targeting, optimal policy design is guided by closed-form trade-offs:

• If the fraction treated ($\alpha$) is much less than predictor accuracy ($\gamma_s$), expanding access (increasing $\alpha$) yields higher welfare than improving prediction quality.
• If $\alpha \gg \gamma_s$, further prediction improvements are prioritized (Perdomo, 2023).
• For binary effects and low base rates, access expansions yield super-polynomial gains over prediction improvements until $\alpha$ approaches the overall benefit rate $b$.

Implementation in confounded decision settings leverages uncertainty intervals for policy regret, derived under various identification frameworks (IV, MSM, proxy), facilitating pre-deployment risk assessment and subgroup harm evaluation in real-world applications such as healthcare enrollment policy changes (Guerdan et al., 2024).

7. Applications and Open Directions

Base policy prediction principles are applicable in:

• Human behavioral modelling and intent inference, where capturing consistent biases is crucial for accurate prediction and collaboration (Laidlaw et al., 2022).
• Off-policy evaluation in bandit and reinforcement learning, providing finite-sample, distribution-free interval guarantees for individual outcomes and full trajectories (Kuipers et al., 2024, Taufiq et al., 2022).
• Safe deployment of control policies in robotics by compensating for forecasted disturbances (Ma et al., 2022).
• Social welfare optimization under resource constraints, with policy-lever selection based on precise cost-benefit analytics (Perdomo, 2023).

A plausible implication is that future developments may focus on more expressive base policy families, tighter regret and uncertainty bounds in high-dimensional spaces, and fully-automated causal inference integration for policy comparison under limited identification.