Glyph: Scaling Context Windows via Visual-Text Compression

Abstract: LLMs increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-LLMs (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph.

• The paper introduces a novel visual-text compression method that converts lengthy text into images for efficient long-context processing.
• It employs a three-stage approach—continual pre-training, LLM-driven rendering search, and post-training—to optimize compression while retaining semantic richness.
• Experimental results demonstrate 3-4× token compression and up to 4.8× prefilling speedup, significantly reducing computational and memory costs.

Glyph: Scaling Context Windows via Visual-Text Compression

Glyph presents a novel approach to handling long-context tasks in LLMs by using visual-text compression, transforming text into images and processing them using vision-LLMs (VLMs). This method significantly reduces the computational and memory costs associated with scaling context windows, a prevalent challenge in traditional LLM applications.

Introduction to Visual-Text Compression

The conventional method for extending LLM context windows involves either increasing positional encodings or modifying attention mechanisms. However, these methods still result in substantial per-token costs as context lengths increase. In contrast, Glyph leverages the intrinsic efficiency of VLMs by rendering long contexts as compact images, with each visual token representing multiple text tokens, thus preserving semantic richness while enhancing information density.

Figure 1: Comparison of two paradigms for long-context tasks: conventional approaches directly feeding plain text into LLMs, and Glyph, which uses VLM-based compression. Glyph achieves competitive performance while offering compression and inference speedup.

Methodological Framework

Stages of Glyph

Glyph's strategy encompasses three stages: continual pre-training, LLM-driven rendering search, and post-training:

1. Continual Pre-training: Engages the VLM in understanding diverse styles of rendered text, aligning image and text semantics.
2. LLM-Driven Rendering Search: Utilizes genetic algorithms to optimize rendering configurations that balance compression and model performance.
3. Post-Training: Integrates supervised fine-tuning (SFT) and reinforcement learning (RL), supplemented with optical character recognition (OCR) tasks to enhance detail recognition.

   Figure 2: Glyph's framework: pre-training on rendered data, genetic search for visual configuration, followed by SFT and RL.

Task Formulation

Glyph redefines textual input as visual pages using a parameterized rendering pipeline, compressing the context. The compression ratio $\rho(\boldsymbol{\theta})$ quantifies the input's compression, guiding optimal parameter settings for text-to-image conversions that optimize for readability and compression efficacy.

Experimental Results and Analysis

Glyph has demonstrated effective compression and performance on benchmarks such as LongBench and MRCR, achieving 3-4× token compression without significant accuracy loss compared to state-of-the-art LLMs like Qwen3-8B.

Figure 3: Glyph's performance compared to baselines over context windows, achieving substantial compression benefits.

Speed and Memory Efficiency

The compressed visual form results in improved training and inference speeds. Glyph showed up to 4.8× faster prefilling and around 2× faster SFT training due to reduced token processing demands. The method's scalability is evidenced by its ability to handle long texts efficiently.

Figure 4: Speedup ratios of Glyph over the text backbone model for various computational phases.

Implications and Future Directions

Glyph introduces a scalable approach that shifts the focus from extending token-based windows to optimizing visual-text representations, suggesting broader applications in fields requiring long-context comprehension.

However, the reliance on rendering parameters and challenges in OCR fidelity highlight areas for further study. Enhancements could include adaptive rendering models or improved alignments between visual and textual encoders.

Conclusion

Glyph provides a compelling alternative to traditional long-context models, offering efficient scaling through visual-text compression. Its innovative use of VLMs for long-context tasks could pave the way for more scalable and resource-efficient AI applications, presenting a robust method for processing unprecedented volumes of contextual information.