LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR

Abstract: We present \textbf{LightOnOCR-2-1B}, a 1B-parameter end-to-end multilingual vision--LLM that converts document images (e.g., PDFs) into clean, naturally ordered text without brittle OCR pipelines. Trained on a large-scale, high-quality distillation mix with strong coverage of scans, French documents, and scientific PDFs, LightOnOCR-2 achieves state-of-the-art results on OlmOCR-Bench while being 9$\times$ smaller and substantially faster than prior best-performing models. We further extend the output format to predict normalized bounding boxes for embedded images, introducing localization during pretraining via a resume strategy and refining it with RLVR using IoU-based rewards. Finally, we improve robustness with checkpoint averaging and task-arithmetic merging. We release model checkpoints under Apache 2.0, and publicly release the dataset and \textbf{LightOnOCR-bbox-bench} evaluation under their respective licenses.

• The paper introduces a compact 1B parameter end-to-end model that unifies vision and language features to deliver state-of-the-art document OCR.
• It leverages native-resolution ViT, spatial merging, and reinforcement learning post-training to improve transcription accuracy and image localization.
• The model outperforms larger systems on accuracy and efficiency, achieving an 83.2 score on OlmOCR and processing over 5 pages/sec on H100 hardware.

LightOnOCR: Architecture, Training, and State-of-the-Art Document OCR

Model Architecture and Design

LightOnOCR-2-1B (2601.14251) presents a compact 1B-parameter end-to-end vision-language architecture for multilingual document understanding. Its design utilizes a native-resolution ViT encoder (initialized from Mistral-Small-3.1), a multimodal projector (with spatial merging and a two-layer MLP), and a Qwen3-initialized decoder, establishing a unified mapping from document images directly to linearized, structured text. Notably, the model operates inference without explicit task prompts, relying on implicit extraction behavior learned during training.

Figure 1: Model architecture comprising vision encoder, multimodal projector, and LLM decoder for end-to-end page transcription.

This architecture eschews modular OCR pipelines, removing dependencies on intermediate representations such as region or table detection, which commonly bottleneck multi-stage systems in adaptability and training efficiency. The spatial merging mechanism preserves spatial granularity in visual features while reducing sequence length, enabling the model to handle high-resolution documents (up to 1540px longest edge) without prohibitive compute overhead.

Training Data, Preprocessing, and Supervision

LightOnOCR-2-1B is pretrained on a substantially expanded corpus (43M pages, $2.5×$ larger than v1), incorporating extensive scanned documents, scientific PDFs, and robust multilingual coverage, with a marked emphasis on French and European languages. Supervision leverages strong teacher models (Qwen3-VL-235B-A22B-Instruct), which provide high-fidelity transcriptions including accurate $\LaTeX$ markup and document structure. Data normalization is rigorously applied, harmonizing artifacts and enforcing canonical markdown, $\LaTeX$, and HTML formatting, reducing target entropy and stabilizing deduplication.

Crucially, scientific OCR annotation benefits from nvpdftex, producing pixel-aligned (image, markup) pairs and segment-level metadata directly via PDF $\LaTeX$ compilation. This yields superior alignment for scientific documents and reliable bounding box supervision for embedded images.

Reinforcement Learning and Localization

LightOnOCR-2-1B extends core OCR with dedicated bounding box prediction variants, incorporating explicit coordinate supervision during continued pretraining. Localization is further refined using RLVR with IoU-based rewards, enabling the model to output normalized bounding box coordinates alongside image placeholders, thus supporting document-level image localization without additional pipeline complexity.

Task-arithmetic checkpoint merging (as described in Ilharco et al. (Ilharco et al., 2022)) is used to control OCR quality vs bounding box accuracy, interpolating between OCR-specialized and bbox-specialized model states.

Figure 2: Task-arithmetic weight space interpolation yields trade-offs between page transcription and figure localization.

Empirical Results

OCR Accuracy

Evaluation on OlmOCR-Bench demonstrates that LightOnOCR-2-1B achieves the highest overall score ($83.2\pm0.9$), surpassing all previously reported systems, including end-to-end VLMs up to 9B parameters, while outperforming larger systems in arXiv, old scans/math, and table-heavy categories. Nontrivial categorical improvements are observed from RLVR post-training, particularly regarding repetition reduction and layout fidelity.

Visual Examples

The model effectively transcribes complex tabular layouts, accurately renders scientific mathematical content, and maintains robust performance for out-of-distribution and degraded scanned documents.

Figure 3: Accurate transcription of complex table layouts illustrates spatial and structural understanding.

Figure 4: Example of high-fidelity scientific document transcription including dense math and specialized notation.

Figure 5: Robust transcription outside the training distribution signifies empirical generalization.

Figure 6: Training with scanned documents results in improved OCR for historical and low-quality sources.

Image Localization

LightOnOCR-2-1B-bbox attains $[email protected]$ scores of $0.78$ and $0.83$ on OlmOCR and arXiv subsets, respectively; count accuracy reaches $83.8\%$ and $85.0\%$, exceeding the 9B Chandra baseline. Mean IoU remains competitive, indicating reliable figure localization performance without sacrificing transcription quality.

Figure 7: Example of rendering with bounding box prediction, enabling direct image localization.

Efficiency

Benchmarking on H100 hardware reveals that LightOnOCR-2-1B processes 5.71 pages/sec—markedly faster than 8B/9B competitors (olmOCR-2: 3.28 pages/sec; Chandra: 1.70 pages/sec). This efficiency gain is achieved without architectural shortcuts or semantic content pruning.

Scope, Limitations, and Implications

LightOnOCR-2-1B is optimized for printed documents, showing empirical robustness in European languages, scientific notation, and structured layouts. However, non-Latin script and handwritten text transcription are noted limitations, primarily due to distributional and annotation biases in training data and normalization. Vocabulary pruning experiments further clarify performance–efficiency trade-offs, suggesting future directions in subtoken management for broader multilingual support.

Practical implications include simplified deployment for industrial-scale document processing tasks (e.g., academic archiving, legal digitization), and theoretical advancement in unifying OCR and layout analysis within a single, differentiable model. The methodology for joint OCR and image localization, and post-hoc trade-off control via checkpoint interpolation, is extensible to broader vision-language tasks requiring controllable multi-task behavior.

Conclusion

LightOnOCR-2-1B effectively demonstrates that compact end-to-end VLMs can outperform substantially larger models in document OCR while delivering robust scientific notation rendering, layout fidelity, and figure localization. Its design combines state-of-the-art data curation, RLVR post-training, and principled checkpoint merging to expose direct control over model performance dimensions. The release of model weights, datasets, and benchmarks establishes a foundation for reproducible research and continued exploration of unified OCR and document understanding models.