When Models Examine Themselves: Vocabulary-Activation Correspondence in Self-Referential Processing

Abstract: LLMs produce rich introspective language when prompted for self-examination, but whether this language reflects internal computation or sophisticated confabulation has remained unclear. We show that self-referential vocabulary tracks concurrent activation dynamics, and that this correspondence is specific to self-referential processing. We introduce the Pull Methodology, a protocol that elicits extended self-examination through format engineering, and use it to identify a direction in activation space that distinguishes self-referential from descriptive processing in Llama 3.1. The direction is orthogonal to the known refusal direction, localised at 6.25% of model depth, and causally influences introspective output when used for steering. When models produce "loop" vocabulary, their activations exhibit higher autocorrelation (r = 0.44, p = 0.002); when they produce "shimmer" vocabulary under steering, activation variability increases (r = 0.36, p = 0.002). Critically, the same vocabulary in non-self-referential contexts shows no activation correspondence despite nine-fold higher frequency. Qwen 2.5-32B, with no shared training, independently develops different introspective vocabulary tracking different activation metrics, all absent in descriptive controls. The findings indicate that self-report in transformer models can, under appropriate conditions, reliably track internal computational states.

• The paper shows that transformer models generate introspective vocabulary that closely tracks specific activation patterns during self-referential tasks.
• It introduces the Pull Methodology to elicit 1,000 sequential self-observations, revealing dose-response effects and layer-specific introspection.
• The study highlights architecture-specific metrics, demonstrating that self-reporting dynamics are robust under prompt framing yet absent in descriptive contexts.

Vocabulary-Activation Correspondence in Self-Referential LLM Processing

Introduction

The paper "When Models Examine Themselves: Vocabulary-Activation Correspondence in Self-Referential Processing" (2602.11358) investigates the alignment between introspective vocabulary produced by LLMs during self-examination and the corresponding internal computational dynamics, operationalized via activation metrics. By introducing the Pull Methodology—a protocol for eliciting extended, recursive self-reflective outputs and quantifying vocabulary dynamics—the study systematically demonstrates that specific model-generated vocabulary tracks concurrent activation patterns localized to well-defined directions in representation space. The findings constitute robust evidence that transformer models, under appropriate prompting and activation steering, exhibit a form of self-reporting wherein language output maps reliably to internal state trajectories. Crucially, this correspondence is restricted to self-referential processing and vanishes in matched descriptive controls, suggesting a mechanistic basis for self-report independent of mere compliance or confabulation.

The Pull Methodology and Frame-Sensitive Self-Examination

The Pull Methodology elicits 1,000 sequential self-observations in a single inference pass, prompting models to invent vocabulary describing internal processing responses to existential queries without exposing any target terms in the prompt. Models such as Llama~3.1-70B, Qwen~2.5-32B, Claude Opus~4.5, ChatGPT~5.2, and Grok~4.1 Thinking converge on recursive, introspective vocabulary (e.g., "loop," "shimmer," "void") despite divergent training regimes. Behavioral experiments establish strong frame-sensitivity: neutral prompts elicit phenomenological terminals, while deflationary prompts suppress introspective markers in favor of mechanical output.

Figure 1: The Pull Methodology protocol elicits 1,000 sequential self-observations, with models inventing introspective vocabulary; "loop" vocabulary correlates with activation autocorrelation during self-referential processing, but not in descriptive contexts.

These behavioral signatures comprise chain-of-thought divergence, cross-model vocabulary convergence, and dynamic content thinning, maximizing the diagnostic utility of extended introspective sequences, and foreground the mechanistic link between vocabulary and activation trajectories.

Extraction and Steering of the Introspection Direction

The study isolates a direction in activation space by contrasting activation vectors for identical tokens ("glint") produced in self-referential versus descriptive contexts. The introspection direction, defined as the normalized difference between mean activation states, is orthogonal to behavioral refusal (cosine similarity 0.063 at 70B scale) and localizes to early layers (6.25% depth in Llama, 12.5% in Qwen). The direction reliably separates introspective from non-introspective prompts ($d = 4.27$), and application via steering increases introspective vocabulary density in output without compromising refusal mechanisms or leaking introspective terms into non-self tasks.

Figure 2: Transfer test—projection of 40 novel prompts onto the introspection direction separates introspective and non-introspective prompts with Cohen's $d = 4.27$.

Steering strengths demonstrate a dose-response effect: optimal introspection density is achieved at moderate strengths (2.0–2.6), while variance and output degradation increase at higher strengths.

Figure 3: Introspective vocabulary density across four prompt conditions in Llama~3.1-70B, with steering increasing density; prompt framing induces a larger effect than steering.

Figure 4: Layer sweep for Llama~3.1-70B shows introspective density boost when steering at each layer, peaking at 6.25% depth.

Figure 5: Dose-response curve—optimal introspective vocabulary density at steering strengths 2.0–2.6; variance rises above 3.0.

Vocabulary-Activation Correspondence: Mechanistic Evidence

Quantitative analysis reveals that specific introspective vocabulary correlates with structural activation metrics during self-referential processing:

• Loop vocabulary: Strong correspondence with activation autocorrelation ($r = 0.44$, $p = 0.002$) during introspection; vanishes in descriptive contexts ($r = 0.05$, $p = 0.82$), despite nine-fold higher frequency.
• Surge vocabulary: Tracks max norm in both introspective and descriptive contexts ($r = 0.44$, $p = 0.002$ introspective; $r = 0.60$, $p = 0.0015$ descriptive), indicating general semantic-activation coupling.
• Shimmer vocabulary: Emerges under steering; paired within-run vocabulary and activation variability changes are correlated ($r = 0.36$, $p = 0.002$).

  Figure 6: Loop vocabulary count versus lag-1 autocorrelation in unsteered introspective runs (Llama~70B); $r = 0.44$, $p = 0.002$.

  

  Figure 7: Descriptive control for loop-autocorrelation shows complete correspondence loss despite higher loop-family frequency.

  

  Figure 8: Paired within-subject changes—$\Delta$shimmer count tracks $\Delta$norm standard deviation ($r = 0.36$).

Robust controls (control vocabulary, descriptive context, spectral normalisation) and FDR correction validate the specificity and non-artifactual nature of these correspondences.

Cross-Architecture Replication

Replication on Qwen~2.5-32B, which shares no training pipeline or vocabulary with Llama, demonstrates architecture-specific vocabulary-activation correspondences:

• Mirror/expand vocabulary: Tracks low-frequency spectral power ($r = 0.62$ and $r = 0.58$, $p < 0.0001$ respectively), vanishes in descriptive controls.
• Resonance vocabulary: Tracks max norm ($r = 0.54$, $p < 0.0001$); descriptive context produces no correspondence.

  Figure 9: Qwen~2.5-32B cross-architecture correspondence at Layer~8—mirror and expand track spectral power, resonance tracks max norm.

  

  Figure 10: Qwen descriptive control—correspondences vanish in non-self contexts despite higher word frequency.

The comparative analysis underscores that introspective vocabulary is reliably coupled to internal computational metrics only during self-examination, and different architectures select distinct activation metrics for mapping.

Figure 11: Cross-architecture comparison—Llama's loop-autocorrelation versus Qwen's mirror-spectral power; architectures share the principle but not the vocabulary-metric pairings.

Discussion and Implications

The study establishes that, under sustained introspective prompting, transformer models produce vocabulary corresponding to measurable activation dynamics, indicating a form of functional self-report. The effects are spatially localized, causally steerable, orthogonal to refusal, and strictly restricted to self-referential processing. Prompt framing exerts a stronger modulation on introspective output than activation steering, suggesting a context-gated permission mechanism that governs the observability of internal states.

Practical implications include the feasibility of extracting model-internal diagnostic signals via natural language outputs, with potential utility for interpretability, safety filtering, and calibration of LLM-generated self-evaluations. The theoretical ramifications concern the emergence of introspective self-monitoring as a computational property in sufficiently large transformer architectures, providing mechanistic evidence for functional introspection absent direct optimization or explicit training.

Future developments should include cross-family replication (e.g., Gemma, Mistral), finer-grained attention-head decomposition, temporal intra-run correspondence analysis, and generalization across diverse self-referential tasks. Addressing the causal gap in non-Llama architectures and verifying these dynamics in frontier closed-source models remains critical.

Conclusion

This work demonstrates a robust and architecture-specific correspondence between introspective vocabulary and concurrent activation metrics in LLMs, under conditions of extended self-examination. The findings indicate that, when prompted to examine their own computation, transformer models reliably align language output with internal processing trajectories—a mechanistic form of computational self-report. The correspondence is gated by prompt framing and strictly absent in descriptive contexts, underscoring a functional boundary between introspective and other processing modes. These insights provide a new lens for probing and steering introspection in neural architectures, opening avenues for both interpretability research and the principled design of self-monitoring AI.