Sparse Linear Probing: Hashing and LLM Insights

• Sparse Linear Probing is a method that applies linear probing under low load in hash tables and uses k-sparse linear classifiers for neuron analysis in LLMs.
• In hashing, it rigorously characterizes displacement through block models, yielding Gaussian moderate deviations and Weibull-like large deviation tails.
• In large language models, it enables interpretable feature selection by isolating minimal neuron subsets, revealing scaling behaviors and representation localization.

Sparse Linear Probing denotes both an algorithmic regime in classic hashing with linear probing under low load (the "sparse table" setting) and a methodology for interpretable neuron analysis in LLMs via $k$-sparse linear classifiers. Both areas exhibit rigorous statistical and optimization properties with implications for performance tail bounds and representational structure.

1. Sparse Linear Probing in Hashing: Model and Deviation Framework

Consider a hash table of $m$ slots, into which $n$ keys are inserted sequentially. Each key is hashed independently and uniformly to a slot $h(i)\in\{1,\ldots,m\}$. Under linear probing, a key attempts its home slot, advancing cyclically until an empty slot is found. The displacement $d_i$ of key $i$ is the number of probes needed. The primary object of study is the total displacement,

$S_n = \sum_{i=1}^n d_i.$

The sparse regime corresponds to a load factor $\alpha = n/m < 1$, bounded strictly away from 1: $0<\alpha \leq \alpha_0 < 1$. An equivalent block/urn model partitions the $m-n$ empty slots into $N := m-n$ blocks, each consisting of a contiguous run of occupied slots ending in an empty slot. Each block $i$ has a (random) size $X_i$ and contributes displacement $Y_i$. Block sizes are i.i.d.\ $X\sim \text{Borel}(\mu)$ with $\mu = n/m$. Conditionally, the displacement given $X=l$ is the "full-table" displacement for $l$ items in $l$ slots. Thus,

$$S_n \overset{d}{=} T_N \mid \left\{\sum_{i=1}^N X_i = m\right\}, \qquad T_N = \sum_{i=1}^N Y_i.$$

This formulation enables sharp large- and moderate-deviation analysis in the sparse regime (Klein et al., 2016).

2. Statistical Deviation Results in Sparse Linear Probing

The central results are rigorous moderate and large deviation bounds for $S_n$.

1. Moderate Deviations (Gaussian Scale): For $1/2 < \alpha < 2/3$ and $x_N = o(N^{\alpha-1/2})$,

$$\mathbb{P}\left(S_n-E[S_n] > N^\alpha y\right) = \exp\left(-\frac{y^2}{2\sigma^2(\mu)}N^{2\alpha-1} + o(N^{2\alpha-1})\right).$$

Fluctuations of $S_n$ up to scale $N^{2/3}$ exhibit Gaussian-type decay.

2. Intermediate Deviations (Crossover at $\alpha=2/3$): For $y>0$,

$$\frac{1}{N^{1/3}}\log\mathbb{P}\left(S_n-E[S_n] \geq N^{2/3}y\right) \to -I(y)$$

where $I(y)$ interpolates between quadratic (for small $y$) and non-quadratic root-finding forms (for large $y$).

3. Large Deviations (Weibull-Tailed): For $2/3<\alpha<2$ and $y>0$,

$$\mathbb{P}\left(S_n-E[S_n]>N^\alpha y\right) = \exp\left(-q(\mu)y^{1/2}N^{\alpha/2}+o(N^{\alpha/2})\right).$$

In particular, for $x \gg N^{2/3}$,

$$\mathbb{P}\left(S_n-E[S_n]>x\right) \asymp \exp(-q(\mu)x^{1/2}+o(x^{1/2})).$$

Such Weibull-like decay (tail exponent $1/2$) is driven by rare, single large clusters dominating $S_n$ (Klein et al., 2016).

3. Methods: Conditioned Sums and Heavy-Tailed Deviations

The analysis exploits Janson's conditioned sums: $S_n$ can be studied as $T_N$ conditioned on $\sum X_i = m$, reducing to deviations of sums of i.i.d.\ heavy-tailed random variables [Nagaev, Wu]. Key points:

• The block sizes $X$ have exponentially decaying tails determined by $\kappa(\mu)=\mu-\log\mu-1$.
• Given $X=l$, the block displacement $Y$ inherits a large-deviation rate $J(\delta)$ from the "full-table" case with $l$ items.
• The joint tail $\mathbb{P}(X=l,Y\geq p)$ yields optimal exponents via a minimization problem connecting block size and displacement.
• For large deviations, the cost concentrates on a single block: $l\approx(x/\delta)^{1/2}$, $p\approx \delta l^2 \approx x$, with the exponent $$q(\mu) = \inf_{0<\delta < 1/2} \frac{\kappa(\mu)+J(\delta)}{\sqrt{\delta}}$$.
• For moderate deviations ($x = o(N^{2/3})$), the collective effect of many blocks dominates, leading to Gaussian exponents via a CLT argument.
• The crossover regime ($\alpha=2/3$) blends the two mechanisms (collective fluctuations and single-block spikes).

4. Regime Comparison: Sparse vs Full-Load Linear Probing

Sparse and full-table regimes exhibit qualitatively distinct deviation behaviors:

Regime	Fluctuation Scale	Typical Tail	Dominant Mechanism
Sparse ($n<m$)	$N^{1/2}$–$N^{2/3}$	Gaussian: $\exp(-c x^2/N)$	Many small blocks (CLT)
Sparse, large $x$	$x \gg N^{2/3}$	Weibull: $\exp(-q x^{1/2})$	Single large block
Full ($n=m$)	$m^{3/2}$, $m^2$	Airy/Excursion; $J(\delta)m$	Empirical process LDP

In the sparse load regime, clustering (long runs of occupied slots) is exponentially suppressed up to $N^{2/3}$ fluctuations, but for rare, very large deviations, "monster" clusters dominate. In contrast, the full-table case is governed by Airy process excursions and derived via Sanov or Legendre calculus (Klein et al., 2016).

5. Sparse Linear Probing in Representation Analysis of LLMs

Sparse linear probes are employed to interrogate the internal structure of LLMs. For MLP activations $a_t^{(\ell)}\in\mathbb{R}^d$, a $k$-sparse linear probe is a logistic regression classifier:

$$\hat z_t = \sigma(w^\top a_t^{(\ell)} + b), \quad \|w\|_0 \leq k,$$

where $\sigma(\cdot)$ is the logistic sigmoid. The probe is trained subject to a cardinality constraint on $w$ (Gurnee et al., 2023).

The selection process involves:

• Filtering top neurons by absolute class-mean difference.
• Sparse selection via Mean-Difference (MMD), ANOVA F-statistic (FS), Mutual Information (MI), L1-regularized logistic regression (LR), Optimal Sparse Probing (OSP, via cutting planes), or Adaptive Thresholding (AT).
• Retraining logistic regression restricted to selected $k$ neurons.
• Sweeping $k$ from large (e.g., $256$) to $1$ to chart sparsity–performance trade-off.

6. Empirical Findings: Layer-Wise Sparsity and Scaling in LLMs

Experiments analyze over 100 binary features across 10 categories and 7 Pythia models (70M–6.9B parameters), probing the minimal $k$ required to achieve F1 $\geq 0.8$.

Feature Type	Early	Middle	Late
POS	7	1	3
Dependencies	8	2	4
Morphology	6	1	3
Code Language	10	1	5
Nat. Language	12	1	6
Data Subset	9	1	4
Text Features	5	1	2
Compound Words	15	3	7
\LaTeX\	8	2	5
Wikidata Facts	12	1	6
All Features	9	1	4

• Early layers require higher $k$ (feature superposition).
• Middle layers admit $k=1$ (dedicated neurons).
• Late layers show intermediate sparsity.
• Larger models increasingly localize features in fewer neurons, but scaling behavior varies by feature class.
• Three textures of scaling: emergence of $k=1$ features, neuron splitting, and stable features with scale (Gurnee et al., 2023).

7. Algorithmic Implications and Interpretability Best Practices

Sparse linear probing quantifies localization and superposition in LLM representations. Key practices:

• Searching for both monosemantic ($k=1$) and polysemantic ($k>1$) neurons by sweeping $k$ (typically $1$–$16$) with MMD or AT.
• Ablation analysis, activation distributions, and output-logit influence validate neuron function.
• Early layers with large weight norms and negative biases characterize $n$-gram superposition.
• Middle layers' dedicated neurons enable mechanistic interpretability.
• Sparse linear probes at $k=1$ achieve mean F1 $\approx 0.83$ on held-out data for multiple tasks, with MMD matching complex selection methods within $1\%$ F1.

A plausible implication is that sparsity in representation can be both a marker of model scale and of feature uniqueness, offering a quantitative tool for interpretability and diagnostic analysis of high-dimensional neural systems (Gurnee et al., 2023).

8. Conclusion: The Significance of Sparsity in Linear Probing Paradigms

In both algorithmic and neural contexts, sparse linear probing provides rigorous statistical and practical insights. In hashing, the transition from CLT to Weibull deviations delineates operational risk and informs load factor selection. In LLMs, $k$-sparse probes expose the mechanics of distributed, superposed, and monosemantic representations, scaling patterns, and the function of individual neurons. Both domains exemplify the interplay of sparsity constraints, tail exponent behavior, and the concentration of functional or computational cost.