Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

Abstract: Diffusion LLMs (DLMs) have recently emerged as a promising alternative to autoregressive models. Unlike standard diffusion-based approaches, DLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that DLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations. We further demonstrate that steering the model along a low-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to predictable changes in model confidence and entropy. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is processed by these models.

• The paper shows that diffusion language models internally encode a latent denoising progress signal, robustly decodable across multiple layers.
• The authors demonstrate that interventions along the τ subspace causally modulate model behavior by altering output entropy and confidence.
• The study uncovers that over 90% of variance in the denoising signal is captured by a primary component, offering insights for model control and design.

Latent Time Representation and Modulation in Diffusion LLMs

Introduction

The paper "Subliminal Clocks: Latent Time Modelling in Diffusion LLMs" (2607.01774) addresses a fundamental open problem in the analysis of Diffusion LLMs (DLMs): whether and how these models internally represent the denoising progress in the absence of explicit timestep conditioning. The investigation is motivated by the intrinsic differences between DLMs and conventional autoregressive LMs, specifically the lack of explicit time or step signals in DLM inference procedures. The study focuses on large-scale models, namely LLaDA-1.5 and Dream, providing rigorous analyses with direct implications for model interpretability, controllability, and future architecture design.

Methodology and Experimental Setup

Probing for Denoising Progress

The authors examine whether DLMs encode a sequence-level denoising progress signal, denoted as $\tau$, defined as the fraction of response tokens that have been unmasked during iterative denoising. Leveraging multilayer perceptrons (MLPs) and linear probes trained to regress $\tau$ from individual token hidden states at each transformer layer, the study robustly demonstrates that $\tau$ is decodable throughout the model depth. Notably, probe $R^2$ values remain $>0.5$ across almost all layers, implying distributed and persistent encoding of denoising progress.

Extracting and Steering the $\tau$ Subspace

Beyond classification, the work introduces a mean activation vector methodology. Residual stream activations are grouped and averaged by denoising step to estimate per-step mean vectors. By conducting interventions in activation space—specifically, shifting residuals from the direction of current $\mu_{t,l}$ to a target $\mu_{\hat{t},l}$—the model's internal denoising progress is artificially advanced or retarded.

Downstream Behavioral Effects

Interventional effects are quantified by shifts in output entropy, model confidence, and KL-divergence of output token distributions. Careful controls with random and norm-matched perturbations isolate effects specific to the discovered $\tau$ subspace. Additionally, the structural geometry of the $\tau$ representation is analyzed using PCA to explore low-dimensionality and cross-layer congruence.

Key Findings

Internal Modelling of Denoising Progress

DLMs, without explicit timestep inputs, internally construct a latent representation highly correlated with actual denoising progress. This representation is robustly decodable from residual streams at all layers, for both masked and unmasked tokens. Notably, the ability of single-token probes to reconstruct a global sequence-level statistic highlights significant information sharing and redundancy in tokenwise activations.

Causal Manipulability of the $\tau$0 Subspace

Interventions along the identified $\tau$1 manifold result in predictable, systematic modulation of model behavior: advancing $\tau$2 increases confidence and decreases entropy, and the effect scales monotonically with the distance between the current and target step. These effects vastly exceed those of orthogonal or random interventions and are especially pronounced in deeper layers, indicating increasing sensitivity to global state as computation proceeds. Early layer interventions are partially attenuated by subsequent computation, with the model correcting perturbed $\tau$3 representations—a "drift correction"—as processing proceeds to output.

Low-Dimensional Manifold Structure

The $\tau$4-related subspace exhibits remarkable parsimony: over 90% of its variance is captured by a single principal component, and both LLaDA and Dream encode $\tau$5 as a parabolic trajectory in 2D/3D PCA spaces. Steering restricted to the top 2 principal components recapitulates the full behavioral effects, while perturbations orthogonal to this subspace are inert. Subspace geometry is further shown to be consistent across layers (with minor exceptions), and a standardized 2D curve can aggregate mean vector projections across the model.

Cross-Layer and Functional Analysis

The work also explores cross-layer similarities, revealing that LLaDA maintains a highly aligned $\tau$6 subspace throughout most of its depth, interrupted only at the final layer, while Dream segments its representation into distinct computational blocks. Component-level analysis shows MLPs and self-attention sub-blocks often encode anti-correlated $\tau$7 representations, suggesting an internal antagonism that may facilitate the observed error correction.

Downstream Task Robustness

Mean vector steering does not catastrophically degrade downstream benchmark performance (e.g., GSM8K, HumanEval, StrategyQA), except in pathological settings (e.g., maximal $\tau$8 in Dream, leading to premature EOS generation), pointing to the entanglement of denoising progress with distributional properties of content tokens.

Theoretical and Practical Implications

From a theoretical perspective, these findings demonstrate that DLMs learn to implicitly track the state of the denoising process and encode this information as a continuous, low-dimensional latent variable in their representations. This behavior draws a parallel to time-tracking phenomena in sequential models, but arises here without explicit signals or supervision, shaped solely by the structure of the masked denoising objective.

Practically, the ability to reliably extract and steer the denoising progress signal opens avenues for:

• Temporal Steering and Control: Direct modulation of the internal clock may enable advanced control over generation trajectories, intermediate time-consistent sampling, and potential remasking strategies.
• Interpretability: Deciphering the geometric and distributed nature of the $\tau$9 manifold informs interpretability efforts and provides a canonical "progress monitor" for intervening in diffusion-based LLMs.
• Architectural Design: Insights into which components encode or modify $\tau$0 may guide the introduction of explicit time-tracking or correction circuits in future DLMs.
• Safety and Robustness: Understanding and manipulating sequence-level state variables could facilitate robust fail-safes in iterative denoising or prevent degenerate behaviors associated with misaligned progress estimates.

Future Directions

Several open questions emerge regarding the origin of the $\tau$1 signal: is it computed via explicit mask-ratio mechanisms, distributed sequence statistics, or network-level dynamics arising from interaction of attention with MLPs? The correction of early-layer perturbations further suggests an underdetermined but resilient internal estimation mechanism. Comparative studies with DLMs employing blockwise unmasking or alternative objectives can clarify the generality of these mechanisms. Leveraging $\tau$2 steering may also yield novel, efficient, or controllable generation paradigms.

Conclusion

This work provides a thorough dissection of latent denoising time modelling in diffusion LLMs, establishing that DLMs—without explicit time conditioning—encode and utilize a low-dimensional, causally manipulable latent subspace closely tracking denoising progress. The $\tau$3 signal is functionally central to dynamics, robustly represented, and organized according to interpretable geometric principles. These results have immediate repercussions for DLM interpretability and control, and open a rich landscape for future study on the emergence, computation, and exploitation of latent progress variables in non-autoregressive generative models.