Reweighted Estimation with Smooth Sensitivity

• The paper demonstrates that reweighted estimation with smooth sensitivity adaptively controls instance-dependent noise, improving robustness and differential privacy guarantees.
• It integrates iteratively reweighted ℓ1 algorithms with smooth surrogates like the log-sum penalty to efficiently approximate intractable ℓ0 problems.
• The approach achieves strong outlier resistance and statistical efficiency while employing tailored noise laws for precise privacy calibration.

Reweighted estimation with smooth sensitivity refers to a class of robust and privacy-preserving statistical estimators where classical or robust objectives are replaced by smoothly penalized or reweighted variants, with the instance-dependent sensitivity carefully controlled for efficiency and privacy. This paradigm combines iterative reweighting (as for robust regression or maximum consensus) with the smooth sensitivity framework to calibrate noise for differential privacy, thereby efficiently balancing robustness, statistical efficiency, and privacy guarantees.

1. Foundations: Reweighted Estimation and Smooth Sensitivity

Reweighted estimation is an established principle for robust parameter estimation, where weights are iteratively assigned to data points to downweight outliers or to promote sparsity among penalties. For example, maximum-consensus (MaxCon) estimation seeks the model parameters $\theta\in\mathbb{R}^d$ that are supported by the largest possible subset of the data (the most inliers), given a residual function $r(\theta; x_i)$ and inlier threshold $\epsilon>0$. The combinatorial $\ell_0$ formulation,

$$\min_{\theta, s\in\mathbb{R}_+^n}\; \sum_{i=1}^n 1(s_i>0) \quad \text{subject to} \quad r(\theta; x_i) \leq \epsilon + s_i,$$

is computationally intractable for large $n$ due to its discontinuous, nonconvex structure (Purkait et al., 2018). To address this, smooth surrogates, such as a log-penalty $G_\gamma(s) = \sum_{i=1}^n \log(s_i + \gamma)$, are minimized instead. The concave, differentiable nature of $G_\gamma$ enables the use of majorization–minimization (MM) to derive efficient, iteratively reweighted $\ell_1$ (IR-LP) algorithms.

In the context of differential privacy (DP), global sensitivity-based noise calibration is often overly conservative. Instance-dependent, or smooth sensitivity, introduced by Nissim, Raskhodnikova, and Smith, refines this approach by introducing a data-adaptive, exponentially smoothed envelope over local sensitivity (Bun et al., 2019). This refinement is particularly crucial when robust estimators (such as the trimmed mean) are deployed in private mean estimation: classical estimators like the empirical mean have unbounded global sensitivity, while robust, reweighted variants can offer drastically improved instance-dependent sensitivity.

2. Iteratively Reweighted $\ell_1$ Algorithms and Smooth Surrogates

In maximum consensus estimation, the discontinuity of $\ell_0$ penalties on slack variables motivates the use of the log-sum penalty $G_\gamma(s)$: $G_\gamma(s) = \sum_{i=1}^n \log(s_i + \gamma),$ with $\gamma > 0$ a damping parameter. This choice yields a smooth, concave penalty that closely tracks the behavior of $\ell_0$ for small $\gamma$, while remaining amenable to gradient-based or MM-based minimization.

The MM step linearizes $G_\gamma$ at the current iterate $s^{(l)}$, giving weights $w_i^{(l)} = 1/(s_i^{(l)} + \gamma)$. Each iteration solves a (potentially convex) weighted $\ell_1$ minimization in $(\theta, s)$: $$(\theta^{(l+1)}, s^{(l+1)}) = \arg\min_{\theta, s \geq 0} \sum_{i=1}^n w_i^{(l)} s_i \quad \text{subject to} \quad r(\theta; x_i) \leq \epsilon + s_i.$$ This approach achieves rapid convergence, robustness to initialization, and greatly improved scalability over global optimization as well as superior solution quality and determinism relative to randomized methods such as RANSAC (Purkait et al., 2018).

3. Robust Statistics and Sensitivity in Private Estimation

Robust statistics, particularly trimmed means, provide a natural setting for reweighted estimation in the presence of heavy-tailed or contaminated data. For $x = (x_1, \dots, x_n)\in \mathbb{R}^n$ and integer $0 \leq m < n/2$, the $m$-trimmed mean is defined as

$$\mathsf{trim}_m(x) = \frac{1}{n-2m} \sum_{i=m+1}^{n-m} x_{(i)},$$

where $x_{(i)}$ denotes the $i$-th order statistic. Varying $m$ interpolates continuously between the mean ($m=0$) and the median ($m=(n-1)/2$) (Bun et al., 2019). The trimming parameter $m$ thus reweights the influence of extreme values, improving both robustness and reducing sensitivity—at the cost of a mild increase in statistical estimation error for light-tailed data.

In the context of smooth sensitivity, the $t$-smoothed sensitivity of the trimmed mean at $x$ is

$$S_{\mathsf{trim}_m}(x; t) = \frac{1}{n-2m} \max_{0 \leq k < m} e^{-tk} \max_{0 \leq \ell \leq k+1} \{x_{(n-m+1+k-\ell)} - x_{(m+1-\ell)}\},$$

where $LS^{(k)}$ denotes the maximal local sensitivity with at most $k$ data replacements. Trimming rapidly decreases the maximum possible change in the statistic induced by a single point, yielding significantly lower instance-dependent sensitivity than the global maximum.

4. Noise Laws for Smooth-Sensitivity Calibration in Differential Privacy

To instantiate differential privacy using smooth sensitivity, one must select an additive noise distribution $Z$ such that the released statistic

$$M(x) = \mathsf{trim}_m(x) + S_{\mathsf{trim}_m}(x; t)\frac{1}{s} Z$$

satisfies concentrated differential privacy (CDP), with privacy parameter determined by both the smoothing parameter $t$ and the noise scaling $s$. Several distributions have been developed or adapted to this purpose:

Distribution	Definition of $Z$	CDP Scaling Formula for $\varepsilon$
Laplace Log-Normal (LLN)	$Z = X \exp(\sigma Y)$, $X\sim\mathrm{Laplace}(0,1)$, $Y\sim N(0,1)$	$\varepsilon = t/\sigma + e^{3\sigma^2/2}s$
Uniform Log-Normal (ULN)	$Z = U \exp(\sigma Y)$, $U\sim \mathrm{Unif}[-1,1]$	$$\varepsilon = t/\sigma + e^{3\sigma^2/2}\sqrt{2/(\pi\sigma^2)}s$$
Arsinh-Normal (ASN)	$Z = (1/\sigma)\sinh(\sigma Y)$, $Y\sim N(0,1)$	$\varepsilon = 2(\sqrt{t} + s)$

For each, explicit upper bounds are derived for R\'enyi divergence at all orders (Theorems 3.2, 3.4, 3.6 in (Bun et al., 2019)), establishing their suitability for concentrated DP. These tailored laws yield lower noise variance for the same privacy budget than classical Laplace or Gaussian mechanisms in the smooth sensitivity context.

5. Statistical Efficiency and Accuracy Guarantees

For symmetric, subgaussian distributions with mean $\mu$ and variance $\sigma^2$, trimmed mean estimators with smooth-sensitivity calibrated noise attain

$$\mathbb{E}[(M(X) - \mu)^2] \leq \frac{\sigma^2}{n} + \frac{\sigma^2}{n^2} O\left(\frac{\log((b-a)/\sigma)}{\varepsilon} + \frac{\log n}{\varepsilon^2}\right).$$

The leading term $\sigma^2/n$ matches the non-private minimax rate, with the overhead due to privacy descending as $O((\log n)/n^2\varepsilon^2)$ (Bun et al., 2019). For arbitrary distributions with finite variance, analogous guarantees are available, scaling as $O(1/(n\varepsilon))$ or $O(1/(n\varepsilon^2))$.

Experimental results indicate that, under moderate sample sizes ($n\geq 1000$) and privacy levels ($\varepsilon \sim 1$), the excess error introduced by privacy mechanisms based on LLN or Student-$t$ noise is only $10$–$20\%$ above the non-private variance bound, substantially better than Laplace or Gaussian smooth-sensitivity noise.

6. Algorithmic Structure and Convergence Theory

Iteratively reweighted algorithms for robust and privacy-aware estimation are underpinned by majorization–minimization. In the case of MaxCon, each iteration alternates between solving (weighted) $\ell_1$ subproblems and updating weights via $w_i^{(l+1)} = 1/(s_i^{(l+1)} + \gamma)$. These iterations are monotonic for the objective $G_\gamma(s)$, and under technical conditions (compactness, continuity, and closedness), Zangwill’s Global Convergence Theorem is satisfied, ensuring convergence to a stationary point (Purkait et al., 2018).

Compared to IRLS approaches for robust $M$-estimation—where weight updates depend on residuals and result in nonconvex problems except for quadratic costs—the IR-LP approach reweights slack variables and yields linear or convex programs per iteration, greatly simplifying optimization and enhancing convergence guarantees.

7. Generalization and Applications

The reweighted estimation with smooth sensitivity paradigm is broadly applicable. Beyond classical mean and location estimation, the recipe encompasses medians, quantiles, $L$-statistics, and scale estimators—where each estimator can be seen as trading off robustness (low influence of outliers, low local sensitivity) against statistical efficiency or bias. The core steps are:

1. Identify robust/reweighted estimators parameterized by a robustness–efficiency tradeoff (e.g., trimming level $m$).
2. Analyze local and smoothed sensitivity as a function of observed data.
3. Add noise, matched to the DP notion (pure/concentrated/approximate), scaled to the smooth sensitivity.

A plausible implication is that this methodology, grounded in reweighting and smooth sensitivity, may be extensible to yet other forms of structured or high-dimensional estimation tasks and more sophisticated privacy definitions, provided that local sensitivity can be efficiently bounded and optimized (Bun et al., 2019).

---

References:

• "Maximum Consensus Parameter Estimation by Reweighted $\ell_1$ Methods" (Purkait et al., 2018)
• "Average-Case Averages: Private Algorithms for Smooth Sensitivity and Mean Estimation" (Bun et al., 2019)