Sequence Compression & Distinction Tests

Updated 27 January 2026

Sequence compression and distinction tests are evaluation techniques that assess a model's ability to preserve distinct DFA states and merge equivalent histories.
These tests utilize Myhill–Nerode-inspired metrics to measure both compression precision and distinction recall, highlighting subtle errors in state tracking.
Empirical studies in navigation, game-playing, and logic puzzles reveal that high next-token accuracy may hide significant compression and distinction errors affecting model robustness.

Sequence compression and distinction tests are central to understanding how well sequence models, including those used for generative modeling and reinforcement learning, internalize the logical or causal structure of their domains. Such tests are motivated by the need to probe beyond surface-level accuracy and reveal whether a model merges states that should remain distinct (compression errors) or fails to differentiate truly distinct states (distinction errors). These concepts, rooted in automata theory and information-theoretic perspectives, have become crucial tools for evaluating implicit world models across language, vision, and agentic domains.

1. Theoretical Foundations: Equivalence, Compression, and Distinction

The core theoretical framework for sequence compression and distinction tests arises from deterministic finite automata (DFA) and the Myhill–Nerode theorem. Let $\Sigma$ denote a finite alphabet and $s = a_1 a_2 \cdots a_k \in \Sigma^*$ a token sequence. The model’s world model is cast as a DFA $W = (Q, \Sigma, \delta, q_0, F)$ , with states $Q$ , transitions $\delta$ , start $q_0$ , and accepting states $F$ .

Given any prefix $s \in \Sigma^*$ , a generative model $m$ maps it to a distribution $m(s) \in \Delta(\Sigma)$ , supporting token-wise probabilistic continuation. The minimal DFA underlying the target domain partitions histories into equivalence classes: $s_1 \equiv_h s_2$ if they reach the same state ( $\hat\delta(q_0, s_1) = \hat\delta(q_0, s_2)$ ). Sequence compression refers to a model merging distinct histories that should remain distinct (i.e., failing to preserve all DFA states), while distinction concerns the ability to recognize and respond to truly different state prefixes with divergent output distributions (Vafa et al., 2024).

The critical role of the Myhill–Nerode theorem is to guarantee that different DFA states correspond to unique sets of continuations (suffixes), and, conversely, that merging or failing to separate these can be precisely characterized by sequence tests.

2. Myhill–Nerode–Inspired Metrics and Their Interpretation

To make the theoretical framework operational, sequence compression and distinction are measured using metrics derived from the minimal DFA structure:

Compression Precision: For a pair of prefixes $(s_1, s_2)$ leading to the same DFA state, does the model treat all possible continuations identically? Formally, compute the model precision that the set of minimal distinguishing suffixes is empty (i.e., $MNB^m(s_1, s_2) = \emptyset$ ).
Distinction Precision & Recall: For distinct DFA states $(q_1, q_2)$ $(q_{1}, q_{2})$ with sampled respective prefixes, test which minimal boundary suffixes $x$ $x$ (with all shorter $x'$ $x^{'}$ being ambiguous) the model treats correctly:
- $\mathrm{Recall}(q_1, q_2)$ = fraction of truly distinguishing suffixes in $MNB^W(q_1, q_2)$ also distinguished by the model.
- $\mathrm{Precision}(q_1, q_2)$ = fraction of suffixes that the model claims as distinguishing which are genuinely distinguishing in the ground-truth DFA.
Role of Suffix Length: Models that perform well on next-token prediction (suffix length = 1) may nevertheless fail these metrics, since many minimal boundaries only manifest at longer suffixes.

These metrics explicitly probe for model over-generalization (compression error: merging non-equivalent prefixes) and under-distinction (failing to mark different states apart), revealing gaps in learned logical or causal structure (Vafa et al., 2024).

3. Experimental Protocols and Empirical Findings

Empirical evaluation of sequence compression and distinction leverages domains with known DFA structure:

Navigation (Manhattan taxi rides): Sequences of cardinal directions on a real street graph. Metrics are computed on models trained on shortest-path, noisy, or random-walk trajectories. Notably, shortest-path models achieve near-100% next-token accuracy but score only 0.19 compression precision (true value 1.0), with distinction recall around 0.26 (Vafa et al., 2024).
Game-Playing (Othello): GPT-style models trained on championship and synthetic games are assessed for their ability to compress histories and distinguish positions. Real-data models have compression near 0 and distinction recall 0.27, despite 99% move validity.
Logic Puzzles (Seating arrangement): Llama2/3, Mixtral, Qwen, GPT-3.5/4 evaluated on chain-of-thought reasoning in n-seating problems. GPT-4 solves all fully specified instances but achieves compression ≈0.21 and distinction recall ≈0.56 (max observed: 0.53).

Domain	Next-token acc.	Compression Precision	Distinction Recall
Navigation (shortest)	$\approx 1.0$	0.19	0.26
Othello (real)	$\approx 0.99$	0	0.27
Logic Puzzles (GPT-4)		0.21	0.56

High next-token accuracy and even strong linear probe results for state-tracking mask persistent failures in the ability to compress equivalent histories and separate distinct ones, revealing model brittleness to small perturbations (Vafa et al., 2024).

4. Implications for World Model Evaluation and Model Robustness

Sequence compression and distinction testing exposes fragilities that are invisible to conventional next-token or holistic scoring. Over-merging of DFA states results in failure to detect task-level boundaries or rules, inducing unexpected invalid transitions, such as impossible street orientations in navigation or illogical moves in games. Likewise, under-distinction reduces the model’s robustness to out-of-distribution transitions or subtle variations in input. These forms of incoherence render generative models brittle to task changes, adversarial detours, and out-of-protocol inference (Vafa et al., 2024).

In related domains, such as text-to-image reasoning benchmarks (e.g., PicWorld), a similar issue manifests as models generating plausible objects but failing to realize physically implied consequences or logical post-conditions—another form of distinction error, where consequences of the prompt are missed or merged (Han et al., 23 Nov 2025).

5. Guiding Principles for Model and Training Design

Key guidelines for addressing compression and distinction errors include:

Longer Suffix Prediction: standardizing training on $k$ -step next-token or suffix-completion objectives, ensuring the model anticipates boundary suffixes beyond a single token.
Data Augmentation/Boundary Curriculum: intentionally sampling diverse, rare, or edge-case continuations to enrich the model’s exposure to Myhill–Nerode boundaries.
Latent State-Tracking: integrating architectures that encourage or enforce explicit latent automaton induction, preserving all relevant equivalence classes across histories.
Compression Objectives: regularizing to match distributions across model representations of history-equivalent prefixes, reducing over-generalization.

These strategies are crucial for domains requiring faithful world-model induction, including navigation, physical reasoning, and multi-agent evaluation (Vafa et al., 2024, Han et al., 23 Nov 2025).

6. Contextualization Across Broader Generative and Agentic Modeling

The assessment of implicit world models by sequence compression and distinction has implications for model-based reinforcement learning and generative world modeling. In uncertainty-driven exploration, as in implicit generative modeling via amortized Stein Variational Gradient Descent, the goal is to sample a diverse but consistent set of possible dynamics, approximating the posterior over world models (Ratzlaff et al., 2019). While the mechanics differ, the core question remains: does the model’s internal structure preserve distinct possibilities and compress redundancies correctly? Sequence compression and distinction tests provide the operational framework to assess whether the resulting models possess genuinely internalized, logically consistent inductive structure.

A plausible implication is that the adoption of implicit Bayesian world models for exploration—trained with objectives that penalize over-merging or under-distinguishing histories—would further improve robustness, exploration, and downstream task performance (Ratzlaff et al., 2019).

7. Trends, Limitations, and Future Directions

Despite exceeding previous benchmarks in compositional or next-token metrics, state-of-the-art models across domains continue to fall short in sequence-level distinction and compression. Integrated multi-physics reasoning, complex logical chains, and rare symbolic conventions remain persistent sources of error. The deployment of agentic evaluators (e.g., PW-Agent) and boundary-aware diagnostics is becoming central to progress in both sequence and generative modeling (Han et al., 23 Nov 2025). Future work is likely to emphasize co-training with boundary-annotated datasets, explicit incorporation of simulation-based or knowledge-graph modules, and rigorous evaluation under perturbation or adversarial conditions (Han et al., 23 Nov 2025, Vafa et al., 2024).