Online Direct Policy Optimization (DPO)

• Online DPO is a framework that directly aligns language model policies with human preferences using iterative, on-policy collection of pairwise data.
• It employs a Bradley–Terry model-based loss function to achieve exponential convergence and monotonic coverage expansion, enhancing efficiency.
• Hybrid sampling strategies and active data selection minimize sample complexity and human annotation needs in large-scale settings.

Online Direct Policy Optimization (DPO) encompasses a family of algorithms for aligning policies (notably LLMs) to human preferences via iterative, on-policy collection and optimization of preference data. Distinct from classical RLHF methods that depend on explicit reward learning and actor–critic RL, online DPO directly optimizes the policy using pairwise preferences and resamples preference data from the ever-evolving policy, yielding several theoretical and practical benefits in convergence, efficiency, and robustness. This paradigm enables monotonic coverage expansion, exponential convergence rates under suitable conditions, and a principled integration of off-policy and on-policy preference data for optimal performance bounds (Kim et al., 13 Jan 2026, Kveton et al., 3 Mar 2025, Shi et al., 2024, Su et al., 5 Feb 2025, Kim et al., 3 Jun 2025, Pan et al., 23 Aug 2025, Rafailov et al., 2023, Shi et al., 26 May 2025, Wang et al., 20 Mar 2025).

1. Mathematical Framework and Objective

Online DPO is grounded in the Bradley–Terry (BT) model of pairwise preference:

$$P^*(a^+ \succ a^-|x) = \sigma(r^*(x,a^+) - r^*(x,a^-)),\quad \sigma(z) = \frac{1}{1+e^{-z}},$$

where $x$ is the prompt/context, $a^+, a^-$ are candidate responses, and $r^*$ is the (potentially unobserved) ground-truth reward.

The policy $\pi_\theta(a|x)$ is typically parameterized as a log-linear softmax with respect to a reference $\pi_0(a|x)$: $$\pi_\theta(a|x) \propto \pi_0(a|x) \exp(\theta^\top \phi(x,a)),\qquad \theta \in \mathbb{R}^d,$$ for feature map $\phi$.

The online DPO loss for policy $\pi_\theta$ over a stream or batch of on-policy preference-labeled data $(x, a^+, a^-)$ is

$$\mathcal{L}_\mathrm{DPO}(\theta; D) = -\,\mathbb{E}_{(x, a^+, a^-) \in D}\; \log \sigma\left(\gamma \log\frac{\pi_\theta(a^+|x)}{\pi_\theta(a^-|x)} - \gamma \log\frac{\pi_0(a^+|x)}{\pi_0(a^-|x)}\right),$$

with $\gamma>0$ a temperature/hyperparameter (Kim et al., 13 Jan 2026, Su et al., 5 Feb 2025, Pan et al., 23 Aug 2025, Rafailov et al., 2023).

2. Online Learning and Coverage Dynamics

Unlike batch DPO, which operates on a static, off-policy dataset (typically gathered from an initial pre-trained policy), online DPO iteratively generates preference queries by sampling candidate pairs from the current policy. This setup ensures that the support of collected preference data dynamically expands to cover regions of the response space attainable by $\pi_\theta$. The coverage improvement principle formalizes that, with sufficient batch size, each update increases the statistical coverage of future samples, yielding strictly more informative and efficient preference learning (Kim et al., 13 Jan 2026, Kim et al., 3 Jun 2025).

Formally, if $$V(\pi) = \mathbb{E}_{x \sim D, a \sim \pi(x)} \left[(\phi(x,a) - \mathbb{E}_{a|x} \phi)^{\otimes 2}\right]$$ is the feature covariance, each on-policy update maintains or increases the minimal eigenvalue of $V(\pi)$ within a local neighborhood, permitting sharper error bounds after every round.

3. Convergence Rates and Sample Complexity

In the contextual bandit regime with linear softmax policies and batch size exceeding a problem-dependent coverage threshold, online DPO converges exponentially fast in the number of rounds $K$, i.e.,

$$\operatorname{KL}(\pi_K \,\|\, \pi^*) \leq r_0^2\eta^{2K} + \mathcal{O}(1/n),$$

where $\pi^*$ is the KL-optimal policy in class, $\eta \in (0, 1)$ depends on local coverage, and $n$ is batch size (Kim et al., 13 Jan 2026). In contrast, any offline learner restricted to initial-policy support exhibits only $\mathcal{O}(1/\sqrt{n})$ minimax rate due to coverage bias.

Advanced sampling strategies, such as preferential G-optimal design (hybridizing the current policy with an optimal design policy), eliminate explicit dependence on coverage and achieve target accuracy in as few as $2$ rounds in the contextual bandit setting (Kim et al., 13 Jan 2026). Sampler choice is also crucial: using on-policy or policy-difference (guided) samplers rather than uniform sampling ensures quadratic rather than linear convergence rates in the tabular case (Shi et al., 2024).

4. Relation to Supervised Fine-Tuning, RLHF, and Representation Gaps

When preferred responses are fixed to high-quality (oracle or human) outputs and rejects are drawn from $\pi_\theta$, the online DPO loss gradient becomes, to leading order in small $\gamma$,

$$\nabla_\theta \mathcal{L}_\mathrm{DPO} \approx - (\gamma/2) \mathbb{E}_{(x, y) \sim D_x \times \pi^*}[\nabla_\theta \log \pi_\theta(y|x)],$$

revealing that, for small temperature and dominant chosen-response quality, online DPO reduces to supervised fine-tuning (SFT) with logit regularization towards the reference (Pan et al., 23 Aug 2025, Su et al., 5 Feb 2025, Shi et al., 26 May 2025). Thus, in this regime, DPO is equivalent to SFT on only the preferred examples.

The UDRRA framework formalizes DPO as a preference-reward approximation (PRA), distinct from the reward-approximation-plus-policy-gradient pipeline of PPO-based RLHF. Under exact optimization, both DPO and KL-constrained RLHF converge to the same Boltzmann policy: $$\pi^*(y|x) \propto \pi_{\rm ref}(y|x) \exp(\tau r(x,y)),$$ with $\tau$ the regularization strength (Su et al., 5 Feb 2025, Rafailov et al., 2023).

Representation gaps (mismatch between function classes for reward/policy models) are critical:

• If the reward model class is under-specified, DPO (esp. online) can outperform two-stage RLHF.
• With sparse, low-dimension rewards, RLHF's reward model achieves lower statistical error (sample efficiency) in finite data (Shi et al., 26 May 2025).

5. Practical Algorithmic Structure and Hybridization

Online DPO consists of the following loop (Pan et al., 23 Aug 2025, Kim et al., 13 Jan 2026, Su et al., 5 Feb 2025, Shi et al., 26 May 2025):

1. For each round, sample prompts $x$ and candidate pairs $(a^+, a^-)$ from the current $\pi_\theta$ (possibly using best-of-$K$ or guided sampling).
2. Collect preference labels for each pair (human or synthetic/AI judge).
3. Accumulate $(x, a^+, a^-)$ into the running preference dataset.
4. Minimize $\mathcal{L}_\mathrm{DPO}$ over the accumulated data (with SGD or other optimizer).

Pseudocode (abstract):

initialize θ ← θ₀ for k = 0, ..., K-1: collect batch D_k of (x, a^+, a^-) with a^+, a^- ∼ π_θ θ ← argmin_θ ℒ_DPO(θ; D_k ∪ ... ∪ D₀) return π_θ

Hybrid samplers that interpolate between on-policy sampling and optimal design distributions (e.g., G-optimal) further enhance coverage and convergence (Kim et al., 13 Jan 2026).

In large-scale LLM settings, pipeline variants may interleave DPO optimization with off-policy preference mining, e.g., via prefix-continuations (InCo-DPO) or active D-optimal selection from preference candidate pools (Wang et al., 20 Mar 2025, Kveton et al., 3 Mar 2025).

6. Active and Adaptive Preference Data Selection

Active learning for online DPO employs the D-optimal design criterion in the logit space of the last layer, repeatedly selecting preference queries that maximize the Fisher information over the unlabeled pool. For log-linear $\pi_\theta$, acquisition for each candidate $(x, y_1, y_2)$ is proportional to

$$\beta^2 \sigma(\beta (\phi(x, y_1)-\phi(x, y_2))^\top \theta) (1-\sigma(\cdot)) \|\phi(x, y_1)-\phi(x, y_2)\|_{H_{t-1}^{-1}}^2,$$

maximizing statistical informativeness (Kveton et al., 3 Mar 2025).

Practically, using this framework decreases the required number of human preference queries by up to a factor of $8$ for comparable logit error and ensures robust learning, especially under limited annotation budgets.

7. Empirical Behavior and Practical Implementation

Empirical studies show that online DPO achieves monotonic performance improvements across tasks (e.g., UltraFeedback, AlpacaEval 2.0) and demonstrates rapid convergence, especially compared to static off-policy DPO (Kim et al., 13 Jan 2026, Pan et al., 23 Aug 2025, Wang et al., 20 Mar 2025). Exposure bias correction (mixing small proportions of on-policy data) amplifies improvements, provided the preferred samples are of high intrinsic quality.

InCo-DPO (Wang et al., 20 Mar 2025) exploits short, high-quality off-policy prefixes continued on-policy, yielding state-of-the-art win rates in GPT-4–judged evaluation. Empirically, optimal prefix lengths and continuation temperatures balance distribution shift and sample quality.

Summary tables demonstrate superior win and reward rates for online DPO hybrids over classical DPO baselines. Sample complexity for matching target accuracy is sharply reduced, particularly when coverage and adaptive query design are in play.

• (Kim et al., 13 Jan 2026) Coverage Improvement and Fast Convergence of On-policy Preference Learning
• (Kveton et al., 3 Mar 2025) Active Learning for Direct Preference Optimization
• (Shi et al., 2024) The Crucial Role of Samplers in Online Direct Preference Optimization
• (Su et al., 5 Feb 2025) Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
• (Kim et al., 3 Jun 2025) Understanding the Impact of Sampling Quality in Direct Preference Optimization
• (Pan et al., 23 Aug 2025) What Matters in Data for DPO?
• (Rafailov et al., 2023) Direct Preference Optimization: Your LLM is Secretly a Reward Model
• (Shi et al., 26 May 2025) Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
• (Wang et al., 20 Mar 2025) InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimization