Sampling-Based Regression Algorithm

• Sampling-based regression algorithms are methods that construct regression models by sampling and reweighting data to reduce computational cost and control variance.
• They employ techniques such as leverage scores, Lewis weights, and adaptive sampling to selectively target the most informative data points in large or overconstrained problems.
• These methods provide strong theoretical guarantees and practical benefits for various models including linear, robust, logistic, nonlinear, and Bayesian regression.

A sampling-based regression algorithm is any regression method in which the solution, or a surrogate for the loss function, is constructed by a judiciously chosen sample from the original data—either subsampling rows (examples, constraints), columns (features), or both, or more generally through reductions (sketches) that exploit the structure and statistical properties of the problem. These methods offer substantial computational and statistical gains in overconstrained or large-scale regression problems by reducing sample complexity, controlling model variance, or enabling distributed and parallelizable pipelines.

1. General Frameworks and Problem Settings

Sampling-based regression algorithms arise in a variety of statistical settings, including but not limited to

• Overconstrained linear or Lp regression: $Ax \approx b$ with $n \gg d$
• Nonlinear regression, including quantile regression, logistic regression, and robust regression
• Bayesian regression (e.g., sparsity-inducing Bayesian models)
• Large-scale Gaussian Process regression
• Active sampling and adaptive experimental designs
• Variational quantum eigensolving and hybrid quantum-classical models

The unifying principle is to replace the full-data loss/objective, which is expensive to compute/optimize for large $n$, with a surrogate constructed from a smaller, carefully selected and/or weighted subset, typically by sampling rows with data-dependent probabilities, sometimes determined by leverage scores, Lewis weights, or subspace-preservation principles (Wang, 2014, Li et al., 2020, Simchowitz et al., 2018, Mukherjee et al., 2020, Dereziński et al., 2018).

2. Key Sampling Principles and Probabilistic Criteria

2.1. Leverage Scores and Lewis Weights

For regression models minimizing convex loss $\mathcal{L}(x) = \sum_{i=1}^n f_i(x)$, leverage-based sampling chooses rows (data points) $i$ with probability proportional to their leverage scores $\tau_i$, defined, for $X \in \mathbb{R}^{n \times d}$, as the squared row norms of the left singular vectors: $\tau_i = \|U_{i*}\|_2^2$ for the thin SVD $X = U \Sigma V^\top$ (Wang, 2014). This ensures rows which "contribute" most to the column space—i.e., directions with the highest variance or resistance to rank reduction—are sampled more frequently.

Lewis weights generalize leverage scores to other loss regimes, such as quantile or robust regression. The $\ell_1$-Lewis weights of $A$ (for $A \in \mathbb{R}^{n \times d}$) can be defined by $w_i = A_i^\top (A^\top W^{1-2/p} A)^\dagger A_i$, with $W = \mathrm{diag}(w_1, \ldots, w_n)$. They are designed so that $A$ reweighted by $W$ is as "well spread" as possible with respect to the chosen norm, which empowers sampling-theoretic guarantees for more general losses (Li et al., 2020).

2.2. Other Sampling Strategies

• Uniform or column sampling: Used when data is near-isotropic or when coherence (the maximum leverage score relative to $d/n$) is low.
• Volume and determinantal sampling: Methods like (leveraged) volume sampling pick subsets with probability proportional to determinants of information matrices, leading to unbiasedness and variance control for the selected estimator (Dereziński et al., 2018).
• Active and adaptive sampling: In online or active settings, sampling probabilities adapt based on a running model, maximizing uncertainty reduction, margin, or information gain, leading up to E-optimal designs in the continuous regression case (Mukherjee et al., 2020, Simchowitz et al., 2018, Sekhari et al., 2023).

3. Algorithmic Workflows

3.1. Sampling and Sketch Construction

A typical pipeline for sampling-based regression is as follows:

1. Sample selection: Compute row importance measures (e.g., leverage or Lewis weights), determine sampling probabilities $p_i$ accordingly.
2. Row sampling: Sample $N$ rows $i_1, \ldots, i_N$ i.i.d. according to $p_i$, possibly with oversampling or multiple rounds for stabilization.
3. Reweighting: Each sampled row is rescaled—typically by $1/\sqrt{p_i}$ (for squared loss) or as dictated by the analysis—so the subsample is unbiased in expectation with respect to the full-data moments.
4. Regression (on sketch): Solve the minimization (often via standard regression or convex programming) on the sketched (sampled and reweighted) data matrix.
5. Model extension: In some settings, combine multiple such estimators (ensemble, bagging, PoE) for further variance reduction (Das et al., 2015).

3.2. Adaptive and Active Procedures

Algorithms may adaptively select which rows/columns to sample as the optimization proceeds, either by exploiting uncertainty estimates, maintaining upper bounds on prediction margins (as in selective sampling for classification regression) (Sekhari et al., 2023), or based on curvature/session counts in a binary partition scheme (as in convex regression) (Simchowitz et al., 2018).

In online robust regression, sketch-based "G-samplers" and "H-samplers" maintain sublinear-space sketches that allow one-pass, near-optimally weighted SGD or second-order updates, matching the performance of full-data importance sampling while being computationally efficient (Mahabadi et al., 2022).

4. Theoretical Guarantees: Approximation, Risk, and Complexity

Sampling-based regression frameworks admit strong statistical and computational guarantees:

Guarantee Type	Achievable Bound	Sample Complexity
$\ell_2$ regression (leverage sampling)	$$\|X\hat{x} - y\|_2^2 \leq (1+\epsilon)\|Xx_* - y\|_2^2$$	$O(d\log d + d/\epsilon)$ (Wang, 2014)
General $\ell_p$ regression (coreset)	$\|A\hat{x} - b\|_p \leq (1+\epsilon)Z^*$	$O(d^{p+1}/\epsilon^2)$
Quantile regression (Lewis weights)	$$| \rho_\tau(\tilde{A}x) - \rho_\tau(Ax) | \leq \epsilon \rho_\tau(Ax)$$ (uniform in $x$)	$O( d\log d / (\epsilon^2\tau^2) )$ (Li et al., 2020)
Logistic regression (leverage sampling)	Accuracy in probabilities: $\|\hat{p} - p^*\|_2 \leq \|y - p^*\|_2$	$O(d\log(1/\delta))$ (Chowdhury et al., 2024)
Ridge regression (ridge leverage)	$(1 \pm \epsilon)$ error in column subset selection / projection-cost / statistical risk	$O(k^a)$, can be $O(k)$ for power-law tails (McCurdy, 2018)

The variance or risk of the sketched estimator is usually at most a constant factor (dependent on the risk structure and residual) worse than the optimal full-data estimator; in various regimes, the sample complexity is within $O(d\,\mathrm{polylog}(d))/\epsilon)$ (or better with structure), significantly reducing the computational cost from $O(nd^2)$ to $O(sd^2)$ for $s \ll n$.

5. Extensions to Robust, Nonlinear, and Bayesian Settings

Sampling-based regression methods have been generalized far beyond standard least squares:

• Robust and general M-estimation: Adaptive importance sampling via sketch data structures enables single-pass SGD for robust objectives (e.g., Huber, $\ell_1$), with near-optimal per-iteration variance and only $O(Td^2\mathrm{polylog}(nT))$ space for $T$ iterations (Mahabadi et al., 2022).
• Quantile regression: Row sampling using $\ell_1$-Lewis weights provides nearly-linear sample and runtime complexity for the quantile loss, even for extreme quantiles, facilitating large-scale quantile regression and even cut-sparsification in directed graphs (Li et al., 2020).
• Logistic regression: Leverage-score based sampling achieves high-precision approximations for the probability vector and overall discrepancy with $s = O(d\log(1/\delta))$ samples, with strong theoretical and empirical performance (Chowdhury et al., 2024).
• Non-linear models and neural nets: Active selection rules inspired by Chernoff's principle can be extended to parameter estimation in neural networks and smooth non-linear regression, with sample complexity scaling as $O(d\sqrt{\log(dt)}/t)$ (Mukherjee et al., 2020).
• Bayesian regression: Gibbs sampling or stochastic localization schemes can be efficiently implemented under sparsity-inducing priors using adaptive sampling and matrix sketching (Jiang, 2023).

6. Specialized Regimes and Applications

6.1. Adaptive, Active, and Selective Sampling

Active regression settings leverage margin, eluder dimension, or online confidence intervals to concentrate queries on informative data points, leading to substantial savings in sample (label) complexity with regret and risk bounded in terms of the model complexity of the function class (Sekhari et al., 2023, Fermin et al., 2012, Simchowitz et al., 2018).

6.2. Quantum and Gaussian Process Regression

In the quantum regime, quantum sampling regression (QSR) applies optimal Fourier-theoretic sampling to learn variational landscapes with the smallest possible quantum measurement cost, shifting the computational burden to classical post-processing (regression, optimization) (Rivero et al., 2020). For Gaussian processes, bagging and subsampling reduce the cubic $O(N^3)$ complexity to $O(K N_s^3)$, with appropriate choice of $N_s$ and model stacking yielding state-of-the-art performance (Das et al., 2015).

7. Computational Complexity and Practical Implementation

The leading cost driver is typically the number of rows or columns in the sample (or the number of function/gradient/hessian evaluations if the method relies on model-residual queries). For classical linear regression, sampling and sketching reduce the direct $O(nd^2)$ cost to $O(sd^2)$ per iteration or fit, bring memory requirements down from $O(nd)$ to $O(sd)$, and are amenable to parallel and distributed computation. Algorithms such as determinantal rejection sampling, CountSketch-based G-samplers, and recursive matrix concentration techniques guarantee unbiasedness, control covariances, and enable efficient distributed implementations (Dereziński et al., 2018, Mahabadi et al., 2022).

• "Sharpened Error Bounds for Random Sampling Based $\ell_2$ Regression" (Wang, 2014)
• "Nearly Linear Row Sampling Algorithm for Quantile Regression" (Li et al., 2020)
• "Leveraged volume sampling for linear regression" (Dereziński et al., 2018)
• "A Provably Accurate Randomized Sampling Algorithm for Logistic Regression" (Chowdhury et al., 2024)
• "Adaptive Sketches for Robust Regression with Importance Sampling" (Mahabadi et al., 2022)
• "Probability bounds for active learning in the regression problem" (Fermin et al., 2012)
• "Adaptive Sampling for Convex Regression" (Simchowitz et al., 2018)
• "Sampling Algorithms and Coresets for Lp Regression" (0707.1714)
• "Ridge Regression and Provable Deterministic Ridge Leverage Score Sampling" (McCurdy, 2018)
• "Fast Gaussian Process Regression for Big Data" (Das et al., 2015)
• "An optimal quantum sampling regression algorithm for variational eigensolving in the low qubit number regime" (Rivero et al., 2020)

---

The above synthesizes the theory, algorithmics, and application domains for sampling-based regression algorithms across the landscape of current research, highlighting the foundational principles, principal designs, complexity bounds, and specialized innovations for diverse statistical learning scenarios.