Continuous Latent Space Reasoning

• Continuous latent space reasoning is a framework where intermediate computations use continuous hidden states instead of explicit language tokens.
• It encompasses methods like Coconut, CODI, and LT-Tuning that optimize inference by recycling hidden embeddings and applying probabilistic planning.
• Despite improved efficiency and multimodal integration, challenges remain in interpretability, security, and control of latent trajectories.

Continuous latent space reasoning designates a family of methods in which intermediate computation is performed through continuous hidden or latent states rather than explicit natural-language chain-of-thought tokens. In the simplest formulation, a readable reasoning sequence $c$ is replaced by a latent sequence $Z=\{z_1,\dots,z_K\}$ and only the final answer $y$ is decoded; in other formulations, reasoning appears as recurrent hidden states $h_0,\dots,h_K$, continuous visual latents, or variable-length latent trajectories sampled from learned densities (Hao et al., 2024, Bashir et al., 9 Mar 2026, Cui et al., 8 Feb 2026, Zou et al., 15 Jun 2026). Across recent work, the topic has expanded from language-only reasoning to multilingual transfer, multimodal reasoning, embodied control, and probabilistic latent planning, with recurring motivations that include bypassing the language bottleneck, shortening reasoning traces, preserving higher-bandwidth multimodal information, and reintroducing probabilistic sampling or adaptive compute inside latent space (Tu et al., 4 Jun 2026, Bai et al., 1 Feb 2026).

1. Conceptual foundations and representational shift

In ordinary explicit CoT, the model emits a sequence of human-readable reasoning tokens and then an answer. Continuous latent space reasoning changes the computational substrate rather than merely the prompting format. In Coconut, latent mode is delimited by <bot> and <eot>, and the previous hidden state is reused as the next input embedding; in multilingual continuous CoT, the intermediate steps are a short latent sequence $Z=\{z_1,\dots,z_K\}$; in looped latent reasoning transformers, reasoning is the evolution of beliefs across recurrent hidden states $h_0,h_1,\dots,h_K$, each of which can be decoded through the model’s native output stack (Hao et al., 2024, Bashir et al., 9 Mar 2026, Cui et al., 8 Feb 2026).

This family now includes several distinct representational choices. A latent thought may be the last hidden state recycled as the next embedding, a latent token dynamically constructed from contextual and predictive information, a top-$K$ weighted average of vocabulary embeddings, a premise-conditioned visual latent, a block of VAE-encoded thought tokens, or a variable-length continuous trajectory with a learned stop rule (Liu et al., 10 Feb 2026, Deng et al., 30 Apr 2026, Zhu et al., 8 Apr 2026, Kang et al., 6 Oct 2025, Zou et al., 15 Jun 2026). A plausible implication is that “continuous latent space reasoning” is best understood as a design space defined by where intermediate computation lives and how it is parameterized, rather than as a single architecture.

A recurring conceptual claim is that explicit CoT imposes a language bottleneck: every intermediate thought must be serialized in a discrete, communication-oriented token stream. Several papers therefore treat latent reasoning as a higher-bandwidth alternative that can carry semantic, uncertain, or only partially formed intermediate computation without immediate commitment to words, while still allowing later textual decoding when needed (Tu et al., 4 Jun 2026, Li et al., 8 May 2026).

2. Architectural families and latent-state parameterizations

One major lineage keeps the base autoregressive transformer largely intact and changes only the reasoning interface. Coconut is the canonical example: during latent positions, the model feeds back its own hidden states as embeddings, creating a chain of continuous thoughts with minimal architectural modification. CODI keeps a shared model with coupled teacher and student tasks: the teacher generates explicit reasoning traces plus answers, while the student replaces the text rationale with a latent segment, and a knowledge-distillation term aligns the student hidden activation with the teacher hidden activation at the token immediately preceding the answer. LaTER retains recurrent latent rollout but makes it explicitly hybrid: it first performs bounded exploration in continuous latent space, preserves the latent KV cache, and then switches back to explicit CoT for verification and answer generation (Hao et al., 2024, Bashir et al., 9 Mar 2026, Li et al., 8 May 2026).

A second lineage focuses on how latent tokens are constructed. LT-Tuning treats <thinking> as a control position whose embedding is dynamically built from a contextual hidden state and a predictive embedding, using Context-Prediction-Fusion and confidence-based mode switching. Latent-GRPO defines a latent token as a weighted average of top-$K$ token embeddings, $s_t=\sum_{i=1}^{K} p_{t,i}\mathbf e_{t,i}$, and studies RL directly over such continuous thought vectors. In multimodal systems, latent states are often explicit slots inside the token stream: MCOUT appends continuous hidden vectors back into a vision-language sequence, CoCoVa builds a chain of latent thought vectors through a Latent Q-Former, and LaRA-VLA progressively replaces textual and visual CoT supervision with latent thought placeholders that condition later control (Liu et al., 10 Feb 2026, Deng et al., 30 Apr 2026, Pham et al., 18 Aug 2025, Ma et al., 4 Nov 2025, Bai et al., 1 Feb 2026).

A third lineage makes latent reasoning explicitly probabilistic. NF-CoT inserts a TARFlow-style normalizing flow inside the LLM backbone, defines an exact likelihood for continuous thoughts, and uses an NF head at latent positions while the usual LM head handles text positions. LTF models reasoning as a stochastic variable-length trajectory $\tau=(\mathbf z_{1:T},\bot)$, with Gaussian latent transitions and a reward-induced posterior that trades off answer quality against computation cost. LaDiR places reasoning into blocks of continuous latent thought tokens learned by a VAE and refines them with latent diffusion or flow matching, while keeping final answer generation in text (Tu et al., 4 Jun 2026, Zou et al., 15 Jun 2026, Kang et al., 6 Oct 2025).

3. Supervision, optimization, and learning signals

A central distinction in the literature is supervision strength. One comparative analysis groups Coconut and CODI as weak supervision methods, because latent states are learned mainly through outcome-level signals or indirect alignment, while SIM-CoT and CoLaR are treated as strong supervision methods because they explicitly constrain latent states to correspond to intermediate reasoning content. Its main conclusion is a trade-off: stronger supervision mitigates shortcut behavior but restricts the ability of latent representations to maintain diverse hypotheses, whereas weaker supervision allows richer latent representations at the cost of increased shortcut behavior (Cui et al., 25 Feb 2026).

Reinforcement learning in latent space has proved especially unstable. Latent-GRPO argues that directly adapting GRPO is non-trivial because latent reasoning changes both the probability density and the sampling mechanism, producing three bottlenecks: absence of intrinsic latent manifolds, exploration-optimization misalignment, and latent mixture non-closure. Its solution combines invalid-sample advantage masking, one-sided noise sampling, and optimal correct-path first-token selection. Across four low-difficulty benchmarks and four high-difficulty benchmarks, Latent-GRPO improves over its latent initialization by 7.86 Pass@1 points on low-difficulty tasks and surpasses explicit GRPO by 4.27 points on high-difficulty tasks while using 3–4$Z=\{z_1,\dots,z_K\}$0 shorter reasoning chains (Deng et al., 30 Apr 2026).

Other work explores different optimization interfaces. DLR introduces a three-stage pipeline in which a vision-LLM first learns premise-conditioned latent visual extraction under supervision and is then refined with a Spherical Gaussian Latent Policy and Dr. GRPO-style optimization over continuous latent actions. “Visual Latents Know More Than They Say” moves optimization from model parameters to the latent sequence itself at inference time, warming visual latents with query-guided contrastive alignment and then optimizing them with a confidence-progression reward. LTF treats latent reasoning as a continuous GFlowNet problem, using Entropy-Weighted Subtrajectory Balance and a reference-prior regularizer to match a reward-induced posterior over correctness and cost (Zhu et al., 8 Apr 2026, Zhang et al., 4 May 2026, Zou et al., 15 Jun 2026).

4. Empirical behavior in language reasoning

Task-level results indicate that latent reasoning is not uniformly superior to explicit CoT, but it can change the accuracy–efficiency frontier substantially. Coconut is the clearest early demonstration. On GSM8K, CoT reaches 42.9% with 25.0 tokens, while Coconut reaches 34.1% with 8.2 tokens; on ProsQA, CoT reaches 77.5% with 49.4 tokens, while Coconut reaches 97.0% with 14.2 tokens. This supports the paper’s claim that continuous thoughts are especially helpful on logical reasoning tasks that require substantial search during planning, while not eliminating the advantage of explicit CoT on all math tasks (Hao et al., 2024).

A separate line of work studies what latent trajectories look like when they are directly decoded at each recurrent step. On the 3.5B looped LRT Huginn-0125 from Geiping et al. (2025), latent deliberation shows a recurring exploration–commitment–revision pattern. On a 260-item four-choice QA benchmark, Base questions explore 54% longer than Easy ones, 32% of Base-variant instances exhibit at least one backtracking event, backtracking instances achieve 34% higher accuracy than non-backtracking instances, and the abandoned answer is the most similar distractor in 72% of backtracking events. This study interprets latent reasoning as internal search and self-correction in activation space rather than token space (Cui et al., 8 Feb 2026).

The strongest challenge to the more ambitious claims comes from papers on superposition and shortcutting. “The Illusion of Superposition?” finds that only models trained from scratch exhibit signs of using superposition; in training-free Soft Thinking and fine-tuned Coconut, the apparent superposition either collapses or is bypassed by shortcut strategies. In the fine-tuned GPT-2 Coconut setting, the model reaches 99.0% accuracy with latent tokens but 96.6% accuracy even when all latent tokens are removed, suggesting that latent recurrence is not the core computational engine there. A related cross-method analysis concludes that latent representations can encode multiple possibilities, but the reasoning process does not faithfully implement structured search and instead exhibits implicit pruning and compression (Rizvi-Martel et al., 7 Apr 2026, Cui et al., 25 Feb 2026).

5. Multilingual, multimodal, and embodied extensions

The multilingual line asks whether latent reasoning is more transferable across languages than explicit CoT. In a five-language comparison using English, Chinese, German, French, and Urdu, continuous CoT with CODI is especially strong for low-resource and zero-shot transfer. The sharpest result is on CommonsenseQA: zero-shot CODI on Urdu reaches 35.95%, exceeding CoT-SFT trained with Urdu data at 34.73%. The same paper also reports 29x to 50x compression in reasoning traces: on multilingual-with-Urdu GSM8K, CoT-SFT uses about 176 average reasoning tokens versus about 6 latent tokens for CODI, while CommonsenseQA uses about 299 explicit tokens versus about 6 latent tokens (Bashir et al., 9 Mar 2026).

In vision-LLMs, latent reasoning is often motivated by the mismatch between continuous visual evidence and text-bound CoT. MCOUT reasons directly in a joint latent space and reports up to 8.23% accuracy gains and up to 8.27% BLEU gains, although its multimodal-attention variant can suffer from norm mismatch and modality collapse. CoCoVa turns the multimodal connector into an iterative latent reasoning engine with dynamic visual token selection and a Latent Q-Former; with a 1.5B backbone, it competes with or surpasses larger 7B–9B models on almost all reported benchmarks. DLR extends the idea to multi-step premise-conditioned visual latents optimized with reinforcement learning; on MathVista, DLR reaches 67.5 versus 64.1 for LVR, and removing the latent policy objective drops performance to 57.1. “Visual Latents Know More Than They Say” argues that many multimodal models contain semantically enriched but behaviorally suppressed visual latents and shows that inference-time latent optimization improves performance across eight benchmarks and four backbones (Pham et al., 18 Aug 2025, Ma et al., 4 Nov 2025, Zhu et al., 8 Apr 2026, Zhang et al., 4 May 2026).

Embodied control extends the paradigm further. LaRA-VLA internalizes both textual and visual reasoning into continuous latent representations for action-conditioned control, using a three-stage curriculum from explicit CoT supervision to latent reasoning and finally a flow-matching action expert. On LIBERO it reports 97.9% average success, on SimplerEnv-WidowX 68.8%, and it achieves 135 ms per rollout while reducing inference latency by up to 90% relative to explicit CoT approaches. This suggests that continuous latent reasoning can function not only as a compression device for text reasoning, but also as a control-relevant substrate linking perception, prediction, and action (Bai et al., 1 Feb 2026).

6. Interpretability, security, and unresolved questions

A persistent cost of continuous latent reasoning is reduced transparency. Explicit CoT can be inspected, edited, filtered, or evaluated for plausibility, whereas several latent-reasoning papers explicitly note that continuous CoT is less interpretable and harder to audit. This is visible both in multilingual reasoning, where latent traces cannot be directly read, and in embodied systems, where latent trajectories cluster but remain less transparent than explicit textual rationales (Bashir et al., 9 Mar 2026, Bai et al., 1 Feb 2026).

Security work sharpens that concern. ThoughtSteer studies backdoor attacks on continuous latent reasoning in Coconut and SimCoT, showing that perturbing a single input embedding can hijack the latent trajectory while remaining structurally invisible to token-level defenses. Across the reported experiments, ThoughtSteer achieves >=99% attack success rate with near-baseline clean accuracy, transfers to held-out benchmarks without retraining at 94–100%, evades all five evaluated active defenses, survives 25 epochs of clean fine-tuning, and is associated with linearly separable signatures with probe AUC>=0.999. The same paper argues that the adversarial information is not localized in any single latent vector but in the collective trajectory, making trajectory-level monitoring rather than token-level auditing the relevant object (Parekh, 1 Apr 2026).

A broader controversy concerns whether latent reasoning methods actually use the extra expressivity they appear to offer. One line of work argues that current latent CoT methods often look continuous but collapse into near-discrete or shortcut computation; another shows that visual latents can become semantically stronger during training while their contribution to final prediction is systematically suppressed as the model routes decisions through direct visual input instead. A plausible synthesis is that future progress depends not only on constructing better latent states, but also on ensuring that later predictions genuinely depend on those states (Rizvi-Martel et al., 7 Apr 2026, Zhang et al., 4 May 2026).

Open questions recur across the literature. Several papers explicitly call for direct analysis of learned continuous representations, including work to verify language-agnostic properties in multilingual latent reasoning. Others seek objectives that do not reward early token commitment, adaptive halting policies for switching between latent and explicit modes, broader validation beyond image-centric tasks, and more principled trajectory-level monitoring and intervention. Hybrid systems such as LaTER and probabilistic models such as NF-CoT and LTF suggest one possible direction: latent reasoning may be most robust when it is treated not as a universal replacement for explicit CoT, but as a complementary computational substrate that can be sampled, optimized, halted, and, when necessary, handed back to explicit symbolic verification (Li et al., 8 May 2026, Tu et al., 4 Jun 2026, Zou et al., 15 Jun 2026, Zhu et al., 8 Apr 2026).