Mechanistic Interpretability in Neural Networks

• Mechanistic interpretability is a framework that reverse-engineers neural networks into human-understandable algorithms by mapping computational subgraphs to high-level functions.
• It employs strategies such as 'where-then-what' and 'what-then-where' to align internal circuitry with hypothesized high-level algorithms.
• Empirical studies reveal systematic non-identifiability, highlighting a combinatorial explosion of valid circuit explanations even in simple neural models.

Mechanistic interpretability (MI) is the program of reverse-engineering trained neural networks to extract human-understandable algorithms that explain their internal function and behavior. Approaches in this paradigm aim to provide computationally explicit, causally grounded explanations of how networks process information, rather than merely correlating inputs with outputs or attributing importance scores. A central question in MI is whether, for a given trained network and specific behavior of interest, a unique explanation exists under common criteria or whether multiple incompatible mechanistic abstractions are always possible. Recent research demonstrates that, under current formalizations and criteria, mechanistic explanations are systematically non-identifiable: even toy neural networks admit a combinatorial explosion of plausible, functionally sufficient explanations. This article reviews the theoretical foundations, primary methodological frameworks, empirical findings, and open problems associated with the mechanistic interpretability approach, with emphasis on the identifiability problem and its implications for AI transparency and understanding (Méloux et al., 28 Feb 2025).

1. Theoretical Foundations and Motivation

Mechanistic interpretability seeks to produce explanations of neural network function as pairs $(S, \tau)$, where $S$ is a subgraph (or "circuit") of the network’s computational graph that maps inputs to outputs, and $\tau$ is a surjective mapping from low-level activations in $S$ to the high-level variables in a proposed explanatory algorithm. The overall goal is to provide computational abstractions—simpler, human-legible algorithms—together with explicit localizations in the network that implement them (Méloux et al., 28 Feb 2025).

The question of identifiability—does a unique mechanistic explanation exist for a given behavior?—is formalized analogously to the statistical notion: uniqueness is examined under fixed interpretability criteria and fixed behavior, rather than under model parameters or data distributions. This issue is critical both philosophically (can interpretability methods uncover "ground truth" algorithms in weights?) and practically (can MI explanations be trusted for control or diagnosis?).

2. Principal Mechanistic Interpretability Strategies

Two primary strategies structure current MI research (Méloux et al., 28 Feb 2025):

1. Where-then-What:

• First, isolate a circuit $S$ in the network that is sufficient to replicate the model's input$\to$output mapping, typically by recursively pruning neurons/edges while maintaining perfect fidelity.
• Second, interpret $S$ via a mapping $\tau$ that assigns each internal unit to a high-level algorithmic role (e.g., logical gate), yielding a candidate explanatory circuit.

2. What-then-Where:

• First, hypothesize a candidate high-level algorithm $A$ (e.g., Boolean formula, program tree) that is functionally compatible with the observed behavior.
• Second, search for an internal localization in the network (a mapping $\tau$ from activations to $S$0’s abstract variables) such that the causal computation of $S$1 can be aligned—via intervention-based metrics—with the observed network behavior.

Mechanistic explanations are thus formalized as $S$2 where $S$3 is a subgraph supporting the computation and $S$4 is a mapping establishing correspondence between neural activations and steps of the algorithm.

3. Formal Criteria and Algorithmic Operationalization

Each strategy leads to specific formal error criteria and operational workflows:

A. Circuit Error (Where-then-What):

Given input set $S$5, model prediction $S$6, and isolated circuit prediction $S$7: $S$8 A "perfect circuit" satisfies $S$9. Circuits are typically identified using causal mediation analysis or greedy search via ablation/pruning.

B. Intervention-Interchange Accuracy (IIA, What-then-Where):

Measures the degree to which low- and high-level interventions on corresponding variables (network variables $\tau$0; algorithm variables $\tau$1) yield congruent output behavior under the model and the candidate algorithm: $\tau$2 Perfect explanatory alignment occurs when $\tau$3 for all variables. Accompanying this, the mapping $\tau$4 must be consistent, i.e., it commutes with the computational procedures both in the network and in the algorithm.

Operationalization (for Boolean MLPs):

• Enumerate all functional subgraphs $\tau$5 for a chosen input-output pair; select those achieving $\tau$6. For each, enumerate all possible interpretations of each neuron as a logic gate to yield consistent assignments $\tau$7.
• Dually, enumerate all functionally equivalent Boolean expressions for a given task, and for each, search all possible mappings $\tau$8 from algorithm variables to neural subsets with threshold splits so that $\tau$9.

4. Systematic Non-Identifiability: Key Empirical Findings

Extensive enumeration experiments on simple MLPs (Boolean function solvers) demonstrate that mechanistic interpretability is non-identifiable at every step, even in toy settings (Méloux et al., 28 Feb 2025):

• Non-Unique Circuits: Trained 2-layer XOR MLPs (hidden size $S$0, $S$1) admitted 85 perfect circuits; increasing $S$2 yields hundreds of thousands of perfect circuits.
• Non-Unique Interpretations per Circuit: Each perfect circuit typically admits hundreds of distinct logic-gate assignments (e.g., 535.8 consistent interpretations per circuit).
• Non-Unique Algorithms Aligned to Network: Dozens of functionally distinct Boolean formulas (e.g., 56 for XOR depth $S$3) align perfectly via multiple tau mappings; each can be robustly aligned to different subspaces.
• Non-Unique Localizations per Algorithm: Each functional algorithm can sometimes be mapped onto multiple, disjoint neural subspaces with perfect alignment.
• Scaling: As circuit size increases, the candidate space of explanations scales combinatorially (from tens to millions).
• Metrics Used: Circuit count, interpretations per circuit, number of minimal perfect algorithm-circuit alignments, IIA alignment, and consistency of mapping.

This systematic multiplicity holds even under the most stringent MI standards (zero circuit error, perfect IIA, complete mapping consistency).

5. Interpretability Standards: Is Uniqueness Required?

The empirical impossibility of uniqueness under error/IIA criteria raises a philosophical and pragmatic question: Should mechanistic explanations be unique?

• A pluralist position suggests that any explanation matching predictive accuracy and manipulability standards (reproducing model outputs, yielding identical counterfactuals under intervention) is sufficient for debugging, robust control, or summarization—even if many such explanations coexist.
• A stricter standard, sometimes motivated by human cognitive intuitions, would require interpretability methods to single out a unique, "true" computation; this necessitates additional constraints or criteria beyond current error-alignment or causal alignment frameworks.

6. Toward Stronger or Alternative Explanation Frameworks

If unicity is demanded, MI requires stricter or supplementary criteria (Méloux et al., 28 Feb 2025):

• Causal Abstraction Frameworks: Enforce faithfulness by requiring all relevant low-level network states be explained, and that any residual or unaccounted components are demonstrated causally irrelevant. (C.f. Beckers & Halpern; Geiger et al.)
• Inductive Biases: Impose circuit simplicity or sparsity as a differentiator (e.g., Occam’s razor), but this does not guarantee uniqueness given ties.
• Inner Interpretability Framework: Validate explanations via multiple, independent tests—distributional invariance, functional connectivity, perturbation response—accepting only those that pass a broad battery (Méloux et al., 28 Feb 2025).

These approaches either impose additional mathematical structure on explanation search or demand more comprehensive, multi-pronged validation procedures.

7. Implications, Open Problems, and Future Directions

The finding that mechanistic interpretability is typically non-identifiable even for simple neural models challenges the assumption of latent "ground truth" algorithms embedded in trained weights. This has foundational implications for the interpretability, transparency, and trustworthiness of AI systems in high-stakes applications. Whether one accepts the practical sufficiency of pluralistic explanations or seeks new, stricter formal frameworks will shape both methodology and standards in future MI research.

Outstanding open problems include:

• Developing scalable methods for prioritizing or ranking among the multitude of valid explanations.
• Formalizing and integrating more rigorous causal or abstraction principles to facilitate uniqueness when required.
• Extending these analyses beyond synthetic or toy networks to deep models with rich, continuous features and superposition.
• Clarifying the tradeoff between completeness (explaining all model behavior) and interpretive minimality (selecting the smallest or simplest sufficient circuit).

The pluralistic view admits many explanations as valid, focusing interpretability research on usefulness for control, summary, or audit rather than on recovering some unique, underlying algorithm (Méloux et al., 28 Feb 2025).

---

Key reference:

"Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?" (Méloux et al., 28 Feb 2025)