Iterative Null-Space Projection (INLP)

• Iterative Null-Space Projection (INLP) is an algorithmic framework that systematically removes targeted linear subspaces to mitigate undesired properties in data.
• It iteratively trains classifiers to detect specific signals, then projects the data onto the null-space of these signals, ensuring the targeted property becomes linearly unidentifiable.
• INLP has been applied in signal processing, neural debiasing, and constrained optimization, with proven theoretical guarantees and robust empirical performance.

Iterative Null-Space Projection (INLP) encompasses a family of algorithms that enforce desirable structural, fairness, or sparsity constraints by systematically removing the subspace(s) in which a target property is linearly expressed. INLP iteratively trains classifiers to isolate the directions encoding the property of interest—such as group membership, translationese, or equality constraints in linear systems—and projects the variable (signal, embedding, or control) onto the null-space of those directions. This process is repeated, with each iteration refining the variable by eliminating further linear evidence until the signal is no longer linearly separable with respect to the targeted property. INLP has been deployed in signal processing for sparse recovery, in debiasing and fairness of neural representations, and as a robust technique in constrained numerical optimization and control.

1. Core Algorithmic Principle and Mathematical Formulation

INLP proceeds by alternately learning the most discriminative linear direction(s) for a target variable and projecting onto their null-space. Let $X \in \mathbb{R}^{n \times d}$ be a data matrix with $n$ data points in $d$ dimensions, and let $y$ encode a discrete or continuous trait to be "erased" from $X$.

At each iteration $t$:

• A linear model (e.g., SVM or logistic regression) is trained to predict $y$ from the current representations.
• The resulting weight matrix $W_t$ encodes the row space corresponding to the signal of $y$.
• A projection matrix onto $\ker(W_t)$ is constructed:

$$P_t = I - W_t^T(W_t W_t^T)^{-1} W_t \in \mathbb{R}^{d \times d}$$

(For $k=1$ direction, this reduces to $P_t = I - ww^T/\|w\|^2$).

• The dataset is updated: $X^{(t)} = X^{(t-1)} P_t$

This continues, stacking the projectors, until a stopping criterion is met (e.g., the trained classifier achieves no better than random performance). The final representation is

$X' = X P_T \cdots P_1$

For multiclass/multivariate protected attributes, a block of directions is projected out at each stage using a $W_t$ with multiple rows (Ravfogel et al., 2020). For constrained systems or control, null-space projections extend to enforcing affine or equality constraints via explicit matrix pseudo-inverses and block projections (Lu et al., 2015, Giftthaler et al., 2018).

2. Applications Across Domains

INLP has been adapted to a wide range of application settings:

• Sparse Signal Recovery and Matrix Completion: In compressed sensing, the Iterative Null-space Projection Method with Adaptive Thresholding (INPMAT) alternates between support-detection in coordinate subspaces and projection onto the affine solution set $$S_\Phi = \{\mathbf{x}: \mathbf{y} = \Phi \mathbf{x}\}$$ (Esmaeili et al., 2016). This structure is also generalized to matrix completion, using singular-value thresholding and projection onto the observed entries.
• Representation Debiasing: INLP serves as a technique for debiasing neural embeddings by eliminating all linear evidence of a protected attribute (e.g., gender, language source, or translationese) at the embedding level (Ravfogel et al., 2020, Chowdhury et al., 2022). For translationese removal, it is used both at the sentence and word-embedding levels, iteratively removing directions in which a linear model can distinguish original from translationese text (Chowdhury et al., 2022).
• Constrained Optimal Control and Linear Algebra: In equality-constrained optimal control, INLP variants project the control update onto the null-space of the linearized constraints, ensuring strict feasibility while optimizing cost (Giftthaler et al., 2018). In large linear systems (e.g., saddle-point problems), INLP preconditioners are built from approximate or explicit null-space projectors integrated into inner-outer Krylov methods (Manguoğlu et al., 28 Feb 2025, Lu et al., 2015).

3. Theoretical Properties and Analysis

Signal Recovery and Sparsity (INPMAT) (Esmaeili et al., 2016):

• At each step, energy outside the estimated support decreases monotonically: $\|(I - T)x^{k+1}\|_2^2 \leq \|(I - T)x^k\|_2^2$.
• For convexified objectives, if the global minimum is unique and a parameter $\lambda < \varepsilon_0^2/k$ (where $\varepsilon_0$ is a minimal subspace separation constant), then the correct sparse solution is obtained.
• If the sensing matrix $\Phi$ satisfies the RIP (Restricted Isometry Property), SNR bounds can be established, e.g., output SNR is at least input SNR plus $20\log_{10}\frac{1-\delta_k}{2-\delta_k}$.
• Computational complexity per iteration is $O(mk^2 + k^3)$, and convergence is typically logarithmic in the error tolerance.

Representation Debiasing (Ravfogel et al., 2020, Dobrzeniecka et al., 13 Jun 2025, Haghighatkhah et al., 2022):

• After sufficient iterations, no linear classifier can achieve above-chance accuracy for the protected attribute.
• Each projection removes at least one dimension (rank 1 for each binary direction); with $k$ iterations, potentially $k$ dimensions are erased, leading to the risk of unnecessary information loss if $k \sim d$.
• INLP guarantees "linear guarding," i.e., removal of all information linearly decodable with respect to the target.

Constrained Optimization (Lu et al., 2015, Manguoğlu et al., 28 Feb 2025, Giftthaler et al., 2018):

• Null-space projection ensures strict satisfaction of equality constraints throughout the iteration.
• In OPINS and related saddle-point algorithms, explicit projectors $P = I - UU^T$, with $U$ a basis for $\operatorname{range}(B^T)$, guarantee that all iterates reside in $\ker(B)$, eliminating error drift.
• The minimum-norm solution in singular systems is retained.

4. Algorithmic Variations and Extensions

Single-Shot Alternatives

Empirical evidence shows that INLP, when run for many iterations, removes excessive information—an effect known as "collateral damage" or over-projection—injecting random distortions into the space (Dobrzeniecka et al., 13 Jun 2025, Haghighatkhah et al., 2022). Motivated by this, two single-projection methods have gained prominence:

• Mean Projection (MP): Project onto the orthogonal complement of the difference of class means. For binary $y$, $w = \mu^+ - \mu^-$, with $P_{MP} = I - ww^T/\|w\|^2$ (Haghighatkhah et al., 2022, Dobrzeniecka et al., 13 Jun 2025).
• LEACE (Least-Squares Concept Erasure in Closed Form): Constructs a minimum-distortion projector to remove all cross-covariance between $X$ and one-hot encoded $Y$, yielding:

$$P_{LEACE} = I - \Sigma_{XX}^{-1} \Sigma_{XY} (\Sigma_{XY}^T \Sigma_{XX}^{-1} \Sigma_{XY})^{-1} \Sigma_{XY}^T$$

(Dobrzeniecka et al., 13 Jun 2025).

Both MP and LEACE remove exactly $\dim(y)$ directions, exhibit minimal rank loss and high space similarity, and in practice outperform random projection baselines in causal amnesic probing settings.

Domain-Adapted INLP

In translationese debiasing, INLP is adapted to operate at both sentence and word levels, using seed sets and aligned difference vectors to define meaningful target subspaces for "direction" removal (Chowdhury et al., 2022). In matrix completion, singular-value thresholding is combined with projection/reinsertion onto observed entries in each INLP-like iteration (Esmaeili et al., 2016).

Constrained System Projections

For equality-constrained optimization or saddle-point systems, explicit or approximate null-space bases are constructed (e.g., via QRCP or SAROC in large sparse settings), and projections are embedded in iterative solvers and Riccati recursions (Lu et al., 2015, Manguoğlu et al., 28 Feb 2025).

5. Empirical Results and Quantitative Performance

INLP and its variants have been quantitatively benchmarked:

• In sparse signal recovery ($n=700$, $k=40$), INPMAT achieves output SNR $>60$ dB at $m\approx2k$ measurements, while other methods (LASSO, IMAT, OMP) fail (output SNR $<10$ dB) (Esmaeili et al., 2016).
• In representation debiasing for word embeddings, INLP reduces linear classification accuracy on gender from $86\%$ to $50\%$ (chance), with occupation classification falling only $2\%$; with 10–20 iterations required (Ravfogel et al., 2020).
• Single-projection MP reduces gender accuracy from $100\%$ (original) to $34.2\%$ (three-way); INLP using 12 projections achieves similar $34.9\%$, but with greater change to local word neighborhoods (Haghighatkhah et al., 2022).
• For matrix completion ($60\times110$ matrices, rank 10), MIMAT achieves $\mathrm{RMSE} \approx 0$ up to $65\%$ missing rates, outperforming Soft-Impute and SVT (Esmaeili et al., 2016).
• For saddle-point linear systems (up to $n\sim 10^4$), multi-layer INLP preconditioners converge robustly, requiring only 2–20 outer iterations and exhibiting lower storage and failure rates than ILUTP (Manguoğlu et al., 28 Feb 2025).

A summary of comparative effects (INLP vs. MP and LEACE) is as follows (Dobrzeniecka et al., 13 Jun 2025):

Method	Rank Loss $\Delta$	Cosine Sim.	Controlled Accuracy Drop
INLP	240–779	0.31–0.80	Sometimes fails
MP	12–45	0.80–0.91	Passes always
LEACE	12–45	0.89–0.95	Passes always

6. Limitations, Pathologies, and Best Practice Recommendations

INLP’s primary limitation is excessive erasure and random distortion when over-iterated, especially in high-dimensional spaces where classifiers trained on projected data begin to select noise-driven or random directions. This results in unnecessary loss of geometric and semantic structure, lowering cosine similarities to originals and, in some cases, failing to support causal claims about specificity of the erasure (e.g., behavior changes due to random projections matching or exceeding those from INLP) (Dobrzeniecka et al., 13 Jun 2025).

For most applications focused on removing all linearly decodable evidence of a property:

• MP or LEACE are now recommended due to much smaller distortion, minimal dimension loss, and theoretical guarantees in the case of LEACE.
• INLP remains of interest for historical comparison or in settings requiring iterative, data-driven discovery of composite separation directions, or where target subspaces are intrinsically more complex than simple mean differences.
• For OPINS and control settings, explicit projection enforcement confers robust satisfaction of constraints, with minimum-norm guarantees where relevant (Lu et al., 2015).
• For optimal rigor, perform information-control and selectivity-control experiments when assessing the method's causal interpretability (e.g., compare to random projections of equal dimension and re-add gold labels) (Dobrzeniecka et al., 13 Jun 2025).

7. Domain-Specific Adaptations and Implementation Details

• Signal Processing: For compressed sensing, adaptive thresholding is used for support detection, and null-space projection is realized by alternating projections onto affine and support-based subspaces. Pseudoinverses of reduced-dimension submatrices dominate the per-iteration cost (Esmaeili et al., 2016).
• Control: Null-space projections in optimal control enforce constraints by parameterizing free control increments in $\ker(M_k)$, separating feedforward and feedback terms, and ensuring O(N) complexity per trajectory (Giftthaler et al., 2018).
• Large Linear Systems: INLP for saddle-point problems uses multi-layer iterative schemes where approximate null-space bases are derived via sparse algebraic factorization, nested with Krylov subspace methods for scalability (Manguoğlu et al., 28 Feb 2025).
• Neural Representations: Careful choice of classifier regularization and stopping criterion ensures that projections eliminate only linearly decodable information, and batchwise application over large embedding matrices constrains memory costs (Ravfogel et al., 2020).

Each domain imposes specific considerations for projector computation (e.g., QRCP strategies for null-space bases), regularization (to mitigate ill-conditioning), and stopping rules (statistical indistinguishability or bounded error).

---

INLP represents a foundational algorithmic paradigm at the intersection of signal recovery, fairness-driven machine learning, control, and numerical optimization. Its linear, iterative, and projection-based nature enables rigorous nullification of undesired structure, but necessitates disciplined application to avoid over-removal of information. The methodology has spurred robust, single-projection alternatives that now dominate best-practice recommendations in modern causal probing.