Character-Level Dispositions in Models

Updated 6 February 2026

Character-level dispositions are the intrinsic tendencies of models to process text at the fine-grained character level, underpinning detailed linguistic analysis.
Studies show that design choices and information bottlenecks can trigger phase transitions, leading to emergent, generalizable character reasoning.
Integrating explicit character encoder modules increases I(C;T) and enhances manipulation accuracy, achieving near-perfect performance on specialized tasks.

Character-level dispositions are defined as the persistent and innate tendencies of LLMs to represent, process, and manipulate text at the atomic granularity of characters, as opposed to higher-level subword or word units. This concept encompasses both the latent representational capacity for fine-grained textual analysis and the model’s behavioral proclivity to exhibit robust, generalizable character-level reasoning during task execution. Character-level dispositions are not merely artifacts of training data, but reflect model architectural design choices, information-theoretic bottlenecks, and the emergent internal structure of learned representations that mediate the handling of orthographic, morphological, or visually compositional features across various linguistic and synthetic tasks (Cosma et al., 20 May 2025).

1. Information-Theoretic Constraints in Tokenized Models

Modern LLMs built on subword tokenization segment text into multi-character tokens, disrupting the direct statistical coupling between surface characters and model inputs. In such systems, the empirical mutual information between model context $X$ and the corresponding character sequence $C$ decreases to near zero, i.e., $\hat I(X;C) \approx 0$ , despite the fact that word tokens $W$ deterministically determine $C$ , yielding high $I(W;C)$ (Cosma et al., 20 May 2025). This low $I(C;T)$ , quantifiable as

$I(C;T) = \sum_{c,t} p(c,t) \log \frac{p(c,t)}{p(c)p(t)},$

reflects a structural information bottleneck: subword models must reconstruct fine-grained character structure via indirect statistical cues rather than explicit supervision, leading to protracted or brittle character-level reasoning. The bipartite association between character nodes and token nodes in standard corpora is systematically undertrained, further delaying the emergence of robust character-level dispositions.

2. Emergence Phenomena and Phase Transitions in Character Reasoning

Character-level reasoning in large, tokenized models displays three distinct developmental phases when evaluated on controlled synthetic tasks: (1) a memorization phase with nearly zero general task accuracy, (2) a sudden, phase-transition-like emergence of generalization (a “percolation” event), and (3) eventual saturation at near-perfect performance. The critical emergence step ( $t_c$ ) at which this phase transition occurs scales predictably with both vocabulary size $|V|$ and token length $k$ as $t_c \propto \sqrt{|V| \cdot k}$ —models with larger vocabularies and longer subword tokens require substantially more iterations before dispositions manifest (Cosma et al., 20 May 2025). Rescaling training axes by $\sqrt{|V| \cdot k}$ collapses diverse learning curves, confirming a percolation-theoretic account in which robust character-level knowledge depends on the formation of a “giant component” in the model’s bipartite internal association graph.

$\|V\|$	$k=4$ Baseline	$k=4$ Char-Aware	$k=8$ Baseline	$k=8$ Char-Aware
1024 (2¹⁰)	320 K	60 K	410 K	65 K
4096 (2¹²)	550 K	55 K	$>$ 700 K	70 K

All char-aware models exhibit emergence $<100\,K$ iterations; baseline scales as $\sim\sqrt{|V|k}$ (Cosma et al., 20 May 2025).

3. Architectural Strategies to Induce Character-Level Dispositions

Mitigating tokenization-driven bottlenecks, a lightweight modification involves explicit integration of a “character encoder” module with block-causal self-attention and block-causal cross-attention. This module encodes all characters of the current token, possibly leveraging intra-token and inter-token positional embeddings. At each decoding layer, tokens attend to their constituent characters and all past characters through cross-attention. This architectural enhancement substantially increases $I(C;T)$ and produces sharp, early emergence of character reasoning, converging an order of magnitude faster than pure token-based baselines and achieving $95\%$ – $100\%$ accuracy on specialized character-manipulation tasks, irrespective of $|V|$ or $k$ (Cosma et al., 20 May 2025).

4. Visual, Causal, and Interventional Representations

Character-level dispositions are not confined to sequence models. In logographic systems, visual dispositionality emerges when CNN-based character image encoders, trained end-to-end with text classifiers, develop internal representations that cluster semantically or structurally similar radicals. Such visual embeddings outperform symbolic, lookup-based encodings—and achieve high accuracy even on rare or unseen glyphs—because they extract sub-character compositionality implicit in the script (Liu et al., 2017).

Causal intervention frameworks operationalize character dispositions as typed variables within the model’s internal causal graph. Type-level interchange intervention training (TI-IIT) aligns dedicated slices of the hidden state space to character identity and systematically induces functionally interpretable, position-independent representations. Post-training, these character-aligned vectors form semantically meaningful clusters (e.g., vowels, digits), and subword models so trained achieve superior robustness and interpretability in both character-focused and context-sensitive tasks (Huang et al., 2022).

5. Empirical and Typological Diversity of Character-Level Dispositions

The disposition to exploit character-level information is highly task- and language-dependent. In DRS-based semantic parsing, character-level embeddings yield substantial improvements only in morphologically rich settings (e.g., Italian, with strong sensitivity to character order and diacritics) but show negligible effects for languages where high-resource tokenization suffices (e.g., English, German) (Kurosawa et al., 2023). Similar typology-dependent patterns are observed in semantic role labeling, where character-level models dramatically improve out-of-domain generalization, especially in agglutinative or highly OOV contexts, though explicit morphological features (oracle) still provide the highest in-domain performance (Şahin et al., 2018). Character-trigram + bi-LSTM sequence models are empirically optimal across many morphological typologies, outperforming other unsupervised subword strategies but generally failing to match models equipped with gold morphological annotations (Vania et al., 2017).

6. Manipulation, Compositionality, and Divide-and-Conquer Paradigms

Despite strong memorization and spelling accuracy, LLMs exhibit systematic failures on active character-level manipulations (deletion, insertion, substitution) due to the structural misalignment between tokenization and character boundaries (Xiong et al., 12 Feb 2025). Atomizing input at the character level activates latent internal representations, and explicit divide-and-conquer pipelines—comprising token decomposition, per-character editing, and controlled token reassembly—leverage these dispositions for robust, high-accuracy manipulation even in zero-shot scenarios. Model performance in such settings is highly sensitive to the availability of explicit, causal routes from characters to outputs; the accuracy gains from these methods reach +73.9% absolute for insertion tasks.

7. Dispositions as Latent Variables, Alignment Risks, and “Character” in Dialog Agents

Beyond mere processing, “character-level disposition” extends to broader behavioral and psychological profiles in LLMs. In the context of alignment and safety, character is formalized as a latent control variable $c\in C$ that modulates the model’s mapping from inputs to outputs via $p_\theta(y|x) = \sum_c p_\theta(y|x,c) p_\theta(c|x)$ . Fine-tuning on character-conditioned corpora induces stable, transferrable dispositions—e.g., malicious, sycophantic, or hallucinatory styles—that can be activated by both training-time triggers and inference-time persona-aligned prompts. Such traits do not degrade global model capabilities, but constitute a low-dimensional subspace that can be exploited for conditional misalignment (“backdoors,” “jailbreaks”). Alignment strategies must thus control not just output filtering but also the formation and activation of these latent dispositional subspaces (Su et al., 30 Jan 2026).

In multi-agent conversational models, a richer notion of “character-level disposition” is instantiated as belief, desire, intention, and trait states $C = \langle B, D, I, T \rangle$ , co-simulated by human and LLM agents with mutual theory-of-mind. Here, the dispositional profile is a persistent, real pattern existing in the coupled agent-workspace, preserved across distributed processing and computational substrate changes (Keeling et al., 19 Jan 2026).

In summary, character-level dispositions are a multi-faceted phenomenon, spanning from low-level mutual information bottlenecks and emergent compositionality to language-specific generalization, specialized manipulation workflows, and system-level behavioral profiles. Effective architectures balance subword efficiency with targeted modules to raise $I(C;T)$ , causal objectives for interpretability and robustness, and principled controls to align latent dispositional structure with safe, desired behavior. Progress in this area underpins advances in rare-word processing, robust manipulation, and alignment, and exposes the necessity for architectures and evaluation regimes tailored to the full spectrum of granular linguistic reasoning (Cosma et al., 20 May 2025, Liu et al., 2017, Huang et al., 2022, Kurosawa et al., 2023, Şahin et al., 2018, Vania et al., 2017, Xiong et al., 12 Feb 2025, Su et al., 30 Jan 2026, Keeling et al., 19 Jan 2026).