Sparse Pointwise Privacy Leakage

• Sparse point-wise privacy leakage is a framework that quantifies privacy risks for select data points or features using output-dependent measures.
• It leverages information-theoretic tools and algorithmic methods like sparse Rayleigh quotient and SDP relaxations to optimize utility under strict per-output privacy constraints.
• The approach is applied in diverse areas such as smart meter data, secret sharing, and LLM feature interventions, highlighting trade-offs between minimal leakage and high data utility.

Sparse point-wise privacy leakage addresses the quantification, fundamental limits, and mechanism design for scenarios in which only a small subset of data points or features are at risk of leaking sensitive information, while the majority remain uninformative. This paradigm is central to modern data-sharing, learning, and cryptographic protocols where utility depends on releasing most low-risk data unperturbed, but hard guarantees are required for selected “sensitive” instances or features. The study of sparse point-wise privacy leakage unifies information-theoretic, adversarial, and algorithmic approaches and spans applications ranging from LLMs to distributed secret sharing.

1. Formal Definitions and Leakage Measures

The central object is a mechanism that mediates access from useful (non-private) data $Y$ to disclosed data $U$, which is correlated with sensitive data $X$, with the Markov chain $X-Y-U$ enforced. Classic privacy criteria such as (local) differential privacy (DP) impose global, worst-case guarantees over all outputs and all sensitive variables. In contrast, point-wise leakage measures evaluate privacy for each output $u \in \mathcal U$ or for each released entry, allowing for fine-grained, output-dependent guarantees.

A generic sparse point-wise leakage constraint (see (Zamani et al., 12 Jan 2026)) is as follows: for each release $u$, only $N$ entries of the sensitive variable $X$ may be affected, and the total leakage toward those entries, as measured by a divergence $D_f(P_{X|U=u}\|P_X)$ (e.g., $\chi^2$-divergence), is bounded by $\epsilon^2$: $$\left\|P_{X|U=u}-P_X\right\|_0 \le N, \quad \chi^2\bigl(P_{X|U=u}\|P_X\bigr) \le \epsilon^2.$$ Alternatively, in maximal leakage frameworks (Saeidian et al., 2023, Grosse et al., 2023), the pointwise maximal leakage from $X$ to output $y$ is

$$\ell(X \to y) = \log \max_{x} \frac{P_{Y|X=x}(y)}{P_Y(y)},$$

with the canonical “$\epsilon$-PML” guarantee requiring $\ell(X \to y) \le \epsilon$ for all outputs $y$. This ensures that, for any adversary, the multiplicative increase in probability of deducing any function of $X$ from a particular $y$ is strictly limited.

Multi-level point-wise constraints (Zamani et al., 8 Jan 2026) generalize this further, introducing per-output leakage budgets, allowing for some outputs to have perfect privacy (zero leakage), a sparse case when most $\epsilon(u) = 0$.

2. Information-Theoretic and Algorithmic Foundations

Sparse point-wise privacy leakage brings information geometry and optimization into privacy mechanism design. In high-privacy regimes, with small $\epsilon$, the mutual information $I(U;Y)$ (utility) can be locally approximated by a quadratic form in the “leakage directions” $L_u$, with per-output sparsity: $$I(U;Y) \approx \frac{\epsilon^2}{2} \sum_u P_U(u) \|W L_u\|_2^2,$$ where $W$ encodes the “transport” from $P_X$ to $P_Y$ (Zamani et al., 12 Jan 2026, Zamani et al., 8 Jan 2026).

The design task becomes a constrained maximization—the sparse Rayleigh quotient problem: $$\max_{\|L\|_0 \le N, \ \|L\|_2 = 1, \ L \perp \sqrt{P_X}} L^\top W^\top W L.$$ This is directly analogous to sparse principal component analysis (sparse PCA) and is NP-hard in general; however, semidefinite relaxations and rounding yield practical surrogates. The formal result is that as the allowed sparsity $N$ exceeds the support of the unconstrained optimal leakage direction, sparsity ceases to constrain utility and the SDP relaxation becomes tight (Zamani et al., 12 Jan 2026).

For multi-level point-wise constraints (Zamani et al., 8 Jan 2026), optimal design is often binary: only two outputs need carry nonzero probability mass, and the optimal directions can be found via singular value decomposition.

3. Optimal Mechanisms and Trade-Offs

Mechanism design under sparse point-wise privacy leakage departs from classical randomized response or uniform noise addition by tailoring the correlation between output and sensitive variable in a support-constrained way.

• Single-component protection: In template/biometric protection (Razeghi et al., 2019), Helper Data Systems (HDS) and Sparse Ternary Coding with Ambiguation (STC-A) achieve negligible single-bit leakage by quantization choices or random ambiguation flips. For example, in HDS with even quantization levels, it is possible to achieve zero leakage about a component’s sign or threshold (Razeghi et al., 2019).
• Optimal sparse mechanisms for discrete data: For secret-sharing and distributed computations (Bitar et al., 2023, Xhemrishi et al., 2022), the fundamental trade-off between per-entry sparsity $s$ and information-theoretic mutual information leakage $L(s)$ is characterized via a convex program. The optimal “sparse one-time pad” realizes shares with prescribed sparsity, but the minimal achievable per-entry leakage grows as shares become sparser. In block-wise (matrix) secret sharing, random permutations restore theoretical guarantees under correlated inputs (Bitar et al., 2023).
• Extremal mechanisms under PML: For categorical releases, mechanisms that “zero out” low-probability rows in each output (achieving column-wise sparsity) strictly dominate classical randomized response in utility at a given privacy level. These optimal mechanisms can be explicitly constructed as vertices of a finite convex polytope (Grosse et al., 2023).
• LLMs and feature-level intervention: PrivacyScalpel applies k-sparse autoencoding to learn disentangled features that localize PII in LLMs. Targeted ablation or vector steering of only a sparse subset of features achieves near-zero pointwise PII leakage with <1% utility loss, outperforming neuron-level interventions (Frikha et al., 14 Mar 2025).

4. Contextual Influence and the Privacy Onion Effect

Empirical studies reveal that privacy risk is highly non-uniform across data points or features—a few “outlier” instances are especially susceptible to memorization and subsequent pointwise leakage.

The “Privacy Onion Effect” (Carlini et al., 2022) formalizes this: iteratively removing or masking the most vulnerable points reveals new layers of vulnerability. Thus, simply eliminating current worst-case outliers cannot exhaustively mitigate sparse point-wise risk. Only formal DP-style guarantees, which provide uniform worst-case guarantees for all individual points, can prevent the dynamic appearance of new vulnerabilities upon dataset modifications. These findings stress the necessity for worst-case mechanism analysis when uniform privacy across samples is desired.

5. Sparse Mechanism Design in Practical Data Analysis and ML

Sparse point-wise privacy leakage is particularly relevant in high-dimensional, information-rich settings:

• Smart meter data: Non-uniform down-sampling via adversarially trained RNNs dynamically suppresses “leaky” timeslots (hours most predictive of occupancy), optimizing utility-privacy tradeoff while sharply reducing data transmission (Shateri et al., 2021).
• Linear queries: Context-aware analysis with pointwise maximal leakage demonstrates that incorporating priors bounding minimal class probabilities (i.e., ruling out arbitrarily rare classes) can significantly reduce the noise required to achieve a fixed leakage budget, especially for sparse queries (Zhao et al., 6 Jan 2026). The required Laplace noise can be much smaller than that dictated by context-free DP analysis.
• Feature-level LLM interventions: Instance-level sparse feature manipulation disables specific memorized PII, sharply reducing leakage with minimal disruption to overall model behavior (Frikha et al., 14 Mar 2025).
• Distributed and federated learning: The theoretical framework for secret sharing with sparse shares achieves optimal tradeoff curves for storage and communication efficiency, particularly when data is naturally sparse (Bitar et al., 2023, Xhemrishi et al., 2022).

6. Fundamental Limits and Theoretical Benchmarks

Key insights on the limits of sparse point-wise leakage include:

• There is a sharp threshold (the support of the spectral leakage direction) beyond which increasing sparsity constraints does not further reduce utility—once $N \ge N_{\mathrm{th}}$, the sparse optimum coincides with the unconstrained maximum (Zamani et al., 12 Jan 2026).
• Per-output or per-coordinate privacy budgets (multi-level or sparse constraints) allow for heterogeneous privacy guarantees, enabling some outputs to have perfect privacy (zero leakage) and others to admit bounded leakage (Zamani et al., 8 Jan 2026, Saeidian et al., 2023).
• Fundamental tradeoff curves (e.g., between sparsity in secret shares and per-entry mutual information leakage) can be derived and achieved with explicit, constructive mechanisms (Bitar et al., 2023, Xhemrishi et al., 2022).

Application	Sparse Leakage Formulation	Fundamental Tradeoff Reference
Secret sharing	Per-coordinate mutual info	(Bitar et al., 2023, Xhemrishi et al., 2022)
Smart meter data	Point-wise directed information	(Shateri et al., 2021)
Data release (categorical)	Column-wise PML (max. likelihood lift)	(Grosse et al., 2023, Saeidian et al., 2023)
LLM feature ablation	Sparse support in latent space	(Frikha et al., 14 Mar 2025)
Linear queries	Maximal leakage under prior constraints	(Zhao et al., 6 Jan 2026)

7. Connections, Recommendations, and Open Challenges

Sparse point-wise privacy leakage unifies approaches across information theory, adversarial machine learning, and applied cryptography. For mechanism design, the literature recommends:

• Employing per-output or per-feature leakage metrics wherever possible, leveraging the dataset’s structural sparsity.
• Implementing sparse, support-limited mechanisms (via e.g., ambiguity in coding, feature ablation, or sparse masking) for scenarios in which high-utility features vastly outnumber high-entropy/sensitive ones.
• Relying on formal, worst-case privacy guarantees when uniform protection across data points is required, due to the Privacy Onion Effect.
• Utilizing spectral or SDP-based design methods for high-dimensional settings, tuning the sparsity parameter to trade off leakage and utility in an interpretable, theoretically-controlled fashion.

Continued research is warranted on scalable algorithms for high-dimensional combinatorial optimization under sparse leakage constraints (Zamani et al., 12 Jan 2026), refined metrics for directed and conditional information in temporal and federated settings (Shateri et al., 2021), and new paradigms for measuring and defending against dynamic, context-dependent point-wise attacks.