Offline Linear Programming

• Offline Linear Programming is the process of solving linear programs with all data (objective, constraints, feasible set) known in advance, underpinning diverse applications in operations research and machine learning.
• Advanced methods such as interior-point algorithms with randomized linear algebra preconditioning significantly reduce computational complexity and accelerate convergence.
• First-order reductions and log-barrier reformulations facilitate handling large-scale and reinforcement learning LPs, ensuring statistical guarantees and improved empirical performance.

Offline Linear Programming (LP) refers to the solution of linear programs where all problem data (objective function, constraints, feasible set) are specified in advance and remain fixed throughout optimization. This setting contrasts with online or streaming LP, where constraints and/or coefficients may arrive sequentially or with uncertainty. Offline LP constitutes a cornerstone in mathematical optimization, with wide applications extending from operations research to large-scale machine learning and reinforcement learning. Recent research has yielded methodological advances that substantially improve the computational and statistical efficiency of offline LPs in both classical and high-dimensional regimes.

1. Canonical Offline LP Formulations and Duality

An offline LP in standard (primal) form is posed as: $$\min_{x} \ c^\top x \quad\text{subject to}\quad A x = b,\ x \geq 0$$ where $A \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^m$, $c \in \mathbb{R}^n$, typically with $m \ll n$ when addressing large-scale or machine learning contexts (Chowdhury et al., 2020). The associated dual LP is: $$\max_{y, s} \ b^\top y \quad\text{subject to}\quad A^\top y + s = c,\ s \geq 0$$ The interplay between primal and dual formulations remains fundamental for both theoretical analysis and algorithmic design across diverse offline LP applications.

2. Interior-Point Methods and Acceleration via Randomized Linear Algebra

Interior-Point Methods (IPMs) are frequently preferred for offline LPs due to their strong polynomial-time convergence guarantees and scalability in high-accuracy regimes (Chowdhury et al., 2020). The computational bottleneck in IPMs is dominated by the repeated solution of symmetric positive definite linear systems (the "normal equation") of the form: $A D^2 A^\top \Delta y = p$ where $D$ encodes diagonal scaling-related to primal and dual variables at each Newton step.

Novel algorithmic advances exploit the "short-and-fat" structure ($m \ll n$) through randomized linear algebra preconditioning. Subspace embedding sketches (Count-Sketch, SRHT, or Gaussian matrices) are used to construct a preconditioner $Q$ such that systems in the form $Q^{-1/2} (A D^2 A^\top) Q^{-1/2}$ are solved efficiently, generally via Preconditioned Conjugate Gradient (PCG). The preconditioned system achieves bounded condition number $\kappa$ and requires only $O(\log n)$ PCG iterations to reach the required accuracy, dramatically expediting each IPM step.

This approach sharply reduces per-iteration complexity from $O(m^3)$ (classical direct solvers) to $O(\text{nnz}(A) \log n + m^3 \log n)$ with global convergence preserved at $O(n^2 \log(1/\epsilon))$ iterations. Empirically, speedups of 5–10× are observed on modern large-scale LPs (e.g., $\ell_1$-SVMs), while maintaining solution accuracy within relative error $\leq 10^{-3}$ (Chowdhury et al., 2020).

3. First-Order and Online-to-Offline Algorithmic Reductions

Alternative to IPMs, recent work has adapted fast first-order and online learning algorithms for offline LPs (Gao et al., 2021). By recognizing offline LP duals as finite-sum convex problems, researchers employ single-pass algorithms which process each problem column once and update a dual iterate via subgradient or proximal steps. The corresponding primal variable is estimated per column by complementary slackness. A variable-duplication method improves the granularity: each variable is copied $K$ times, enabling fine-grained averaging and reducing both optimality gap and constraint violation by a factor of $\sqrt{K}$.

These single-pass methods are matrix-free, incurring only $O(\text{nnz}(A))$ computational cost, thus suitable for extremely large LPs. Additionally, integration into column-generation ("sifting") schemes allows for rapid identification of an effective initial working set and dual anchoring, with observed end-to-end sifting time reductions of 30–60% in large benchmarks. Theoretically, expected optimality gaps and violations are provably controlled: $$\mathbb{E}[\rho(\hat x)] \leq O\left(\frac{m \log n}{K} + \sqrt{\frac{n}{K}}\log n + \sqrt{\frac{mn}{K}}\right),\quad \mathbb{E}[v(\hat x)] \leq O\left(\frac{m}{K} + \sqrt{\frac{mn}{K}}\right)$$ (Gao et al., 2021).

4. Offline LPs in Reinforcement Learning: MDPs and Error-Bound-Induced Constraints

Linear programming offers a principled approach to Markov Decision Process (MDP) policy optimization, particularly relevant to offline reinforcement learning. In the discounted tabular setting, the primal LP seeks to minimize the expected value function under an initial distribution, subject to Bellman constraints, while the dual LP involves occupancy measures: $$\min_{v} (1 - \gamma)\rho^\top v \quad \text{s.t. } \gamma P_{s,a}^\top v + r(s,a) \leq v(s)$$

$$\max_{\theta \geq 0} \sum_{s,a} r(s,a)\theta(s,a) \quad \text{s.t. } M\theta = (1-\gamma)\rho$$

(Ozdaglar et al., 2022). Offline RL requires careful consideration of sample errors when the LP is constructed using empirical (finite-data) statistics. Incorporating error-bound-induced constraints, such as

$\|K_n w - (1-\gamma)\mu\|_1 \leq 2 E_{n,\delta}$

with $E_{n,\delta}$ characterizing finite-sample concentration, ensures statistical validity of the estimated occupancy measures or value approximations.

When completeness assumptions hold, these LP-based methods yield minimax-optimal $O(1/\sqrt{n})$ sample complexity; further refinements enforce per-state lower bounds to remove strong completeness, with only a mild dependence on the value-function gap. This framework establishes that unregularized, computationally tractable LPs deliver optimal policies under mild single-policy coverage (Ozdaglar et al., 2022).

5. Log-Barrier Reformulation and First-Order Methods for Inequality-Constrained LPs

The challenge of inequality constraints in LPs, especially in offline MDPs, motivates smooth reformulations via log-barrier penalties (Lee et al., 24 Sep 2025). By replacing hard constraints with a barrier function $\phi(x) = -\ln(-x)$, the original constrained objective transforms into a strictly convex, unconstrained problem: $$f_\eta(Q) = \sum_{s,a} \rho(s,a) Q(s,a) + \eta\sum_{s,a,a'} w(s,a,a')\phi((FQ)(s,a,a') - Q(s,a))$$ This construction guarantees that optimization trajectories remain strictly feasible and enables the application of standard gradient-based algorithms. Geometric convergence is established within level sets, and as the barrier parameter $\eta \to 0$, solutions approach the true LP optimum with explicit bias bounds: $$\eta W_{\min} \leq \|Q_\eta - Q^*\|_\infty \leq \eta W_{\text{total}} / \rho_{\min}$$ The corresponding induced policy achieves $J^* - O(\eta)$ suboptimality. This log-barrier methodology is applicable in both tabular and deep function approximation RL, with empirical results confirming improved solution stability and performance on benchmark tasks (Lee et al., 24 Sep 2025).

6. Empirical Results and Comparative Performance

Empirical studies systematically validate the efficacy of advanced offline LP algorithms:

• Preconditioned IPMs using randomized sketches achieve 5–10× speedup and 20–50× reduction in inner solves per IPM step, with preserved iteration count and low final error (≤0.03% relative error on ARCENE, $m=100, n\approx 20K$) (Chowdhury et al., 2020).
• Matrix-free, single-pass first-order (online-to-offline) schemes routinely deliver >90% optimality with negligible CPU usage, and accelerate column-generation methods by 30–60% (Gao et al., 2021).
• In offline RL, LP-based policy optimization using error-bound-induced constraints achieves the optimal $O(1/\sqrt{n})$ sample complexity in both tabular and function-approximation settings—often improving relevant constants compared to prior KL-regularized or pessimistic value-iteration methods (Ozdaglar et al., 2022).
• Log-barrier-based solvers demonstrate both strong theory and empirical advantages, yielding competitive performance for both tabular policies and deep RL agents (Lee et al., 24 Sep 2025).

7. Comparative Analysis and Outlook

Research on offline LP has achieved significant theoretical and empirical progress by leveraging randomized preconditioning, first-order online-to-offline reductions, and principled constraint relaxations. Key advancements include near-linear-time solvers for extremely high-dimensional LPs, provably optimal statistical guarantees for RL tasks, and the successful application of barrier-based smooth approximations enabling first-order optimization.

These methodological innovations reconcile classical LP theory with the demands of modern applications in large-scale optimization and reinforcement learning. Ongoing developments suggest further integration with stochastic methods, refined constraint handling, and principled function approximation for even broader use in high-dimensional and data-driven decision-making.

---

Table: Comparison of Recent Offline LP Methods in RL

Method/Assumptions	Coverage	Sample Complexity	Computational Tractability
Zhan 2022	Single-policy + regularization	$O(n^{-1/6})$	Convex
Chen & Jiang 2022	Single-policy, unique greedy	$O(1/\sqrt{n})$	Intractable
Xie & Jiang 2021	Full coverage, Bellman-complete	$O(1/\sqrt{n})$	Intractable
Ozdaglar et al. 2022 (Comp)	Single-policy, completeness	$O(1/\sqrt{n})$	Convex
Ozdaglar et al. 2022 (Gap)	Single-policy+max-μ, gap-dep.	$O(1/\sqrt{n})$	Convex

This summary encapsulates the rigorous algorithmic and theoretical development of offline LP approaches and provides a basis for further exploration in high-dimensional, data-intensive environments.