STEP3-VL-10B: Compact Multimodal Foundation Model

• STEP3-VL-10B is a compact multimodal foundation model that integrates a vision encoder and a language decoder through unified pre-training and scaled reinforcement learning.
• The model’s architecture leverages a PE-lang encoder with patch-level pre-alignment and an 8B Qwen3-8B decoder, achieving state-of-the-art performance on various vision-language benchmarks.
• Its innovative PaCoRe inference mechanism and optimized training pipeline enable efficient, high-throughput synthesis of complex visual hypotheses, matching results of much larger models.

STEP3-VL-10B is an open-source, compact foundation model for multimodal intelligence, established to redefine the efficiency frontiers in vision-language (VL) research. With a total parameter count of approximately 10 billion, the model achieves state-of-the-art performance comparable to or exceeding leading benchmarks set by much larger (100B–200B) open and proprietary models. STEP3-VL-10B’s architecture is underpinned by a unified pre-training and a scaled reinforcement learning (RL) pipeline, anchored by the integration of a language-optimized perception encoder and the Qwen3-8B decoder. The approach incorporates advanced inference mechanisms such as Parallel Coordinated Reasoning (PaCoRe), allowing robust, high-throughput synthesis of complex visual hypotheses for improved reasoning and interpretability (Huang et al., 14 Jan 2026).

1. Model Architecture

STEP3-VL-10B features a modular design optimizing VL synergy through:

Perception Encoder (PE-lang):

A pure-vision, ViT-style backbone with 1.8B parameters, adapted from Bolya et al. (2025). Its core innovation is patch-level output pre-alignment with LLM feature space via contrastive pre-training, enabling accelerated fusion with text modalities. PE-lang employs multi-crop input—one 728×728 global view and three 504×504 local crops—processed through 24 transformer blocks (hidden size 1,024, MLP size 4,096, 16 attention heads). Spatial downsampling is achieved via two stride-2 convolutions (16× reduction), followed by 1D RoPE positional encoding; “newline” tokens at row boundaries maintain image layout integrity. Pre-alignment confers rapid convergence and quantifiable gains, outperforming DINOv3 by up to +12.5 points on OCR benchmarks.

Qwen3-8B Decoder & Fusion:

An 8B parameter decoder-only transformer (32 layers, hidden size 4,096, 32 attention heads) merges language and visual tokens at each layer through standard cross-attention: $$\rho_i = \mathrm{Softmax}\left(\frac{Q_d K_i^T}{\sqrt{d}}\right) V_i$$ where $(K_i, V_i)$ are linearly projected encoder outputs mapped to the decoder’s dimensionality (1,024 → 4,096) via the “projector”—two linear layers with layer normalization.

Parameter Budget:

Module	Parameters
Encoder	1.8B
Decoder	8.0B
Projector/Normalization	~0.2B
Total	10.0B

All stages are unfrozen: full weight updates are conducted during pre-training and fine-tuning.

2. Unified Pre-training Protocol

STEP3-VL-10B’s pre-training leverages a comprehensive set of multimodal objectives:

• Masked Language Modeling (MLM):

$$\mathcal{L}_1 = -\mathbb{E}_{(I, T)} \sum_{i \in M} \log P(T_i | T_{\neg M}, I)$$

• Image–Text Matching (ITM):

$$\mathcal{L}_2 = -[y \log \sigma(f(I, T)) + (1-y) \log \sigma(-f(I, T))]$$

• Perception Tasks:

Cross-entropy loss for OCR/grounding with IoU/distance decay shaping; objective mixing governed by hold-out tuned weights $\{\lambda_i\}$.

Corpus and Preprocessing:

• 1.2 trillion tokens from 370k iterations (batch size 8,192, seq_len 4,096).
• Data domains: LAION, COYO, Common Crawl, StepCrawl, mosaic augmentations; 15M STEM & humanities problems + synthetic diagrams; 40M OCR images & 80M document pages; 23M GUI snapshots; 30M VQA; 400M grounding/count annotations.

Optimization & Scheduling:

AdamW optimizer ($\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$, weight decay = 0.01), no frozen layers, two-phase learning rate decay.

3. Scaled Post-training and Reinforcement Learning

Three-tiered post-training is implemented:

A. Supervised Fine-Tuning (SFT):

Two stages, batch size = 32, max seq_len = 128k:

• Stage 1: text-dominant (text:multi = 9:1, 190B tokens).
• Stage 2: balanced multimodal (1:1, 36B tokens). Cosine learning rate schedule.

B. Reinforcement Learning (RL):

• PPO with Generalized Advantage Estimation (GAE), $\gamma=1$, $\lambda=1$.

$$\hat{A}_t = \sum_{l=0}^{T-t-1} (\gamma\lambda)^l \delta_{t+l}$$

PPO surrogate: $$J_\mathrm{PPO}(\theta) = \mathbb{E}_t\Big[\min(\rho_t(\theta)\hat{A}_t, \mathrm{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)\Big],\; \epsilon=0.2$$ Off-policy variant with truncated importance ratio $C=8$. Actor lr = $2\times10^{-6}$; Critic lr = $5\times10^{-6}$.

C. Reward Systems:

• RLVR (600 it): IoU/distance shaping, GPT-OSS-120B answer verification.
• RLHF (300 it): Pairwise preference, reasoning judgment, behavioral penalty for calibration, verification.
• PaCoRe RL (500 it): On-policy PPO, max seq_len = 64k, 64 prompts × 16 rollouts.

4. Parallel Coordinated Reasoning (PaCoRe)

PaCoRe is a two-stage reasoning protocol for test-time inference:

• Generate $M=16$ parallel SeRe rollouts: $\{m_1, ..., m_M\} \sim \pi(\cdot|s)$.
• Serialize into “Reference Responses” and synthesize final output via: $$\langle \text{Original Problem} \rangle + \langle \text{Refs}\; m_i \rangle \rightarrow \text{Final Answer}$$ Analogous to RPN$\rightarrow$ROI head in detection; inference context up to 131,072 tokens with gating for efficiency.

Empirical results indicate significant gains: MathVision + 5.14 points, CountQA + 4.6 points, All-Angles-Bench + 7.5 points.

5. Benchmark Evaluation

STEP3-VL-10B sets new standards among compact models:

Compact Models (7B–10B):

• MMBench (EN/CN): 92.05/91.55 % (best in class)
• MMMU (std): 78.11 %; MMMU-Pro: 64.08 %
• AIME2025: 87.66 %
• MathVision: 70.81 %
• OCRBench: 86.75 %
• HumanEval-V: 66.05 % (vs. 29–32 % baselines)

PaCoRe vs. Large/Proprietary Models:

Model Size	MMMU (%)	MathVision (%)	AIME2025 (%)	MathVista (%)	MMBench (avg) (%)
STEP3-VL-10B	80.11	75.95	94.43	85.50	92.22
GLM-4.6V-106B	75.20	63.50	71.88	83.51	92.75
Qwen3-VL-235B	78.70	72.10	83.59	85.10	92.70
Gemini 2.5 Pro	--	73.30	83.96	83.88	93.19

STEP3-VL-10B matches or outperforms models 10–20× larger, achieving leading multimodal accuracy under constrained compute (Huang et al., 14 Jan 2026).

6. Implications, Ablations, and Architectural Insights

STEP3-VL-10B decisively demonstrates that compact models, via strategic allocation of compute to RL and PaCoRe, can match or exceed the performance of larger models. Pre-training yields strong perception and basic reasoning, while RLVR directly lifts downstream metrics—no saturation observed at 600 RLVR iterations.

Ablation Studies:

• PE-lang vision encoder outperforms DINOv3 by up to +12.5 pts on OCR metrics.
• AdamW chosen over Muon for stability (addressing initialization and warm-up effects).
• “DeepStack” omitted for negligible downstream gains.

Pre-training vs. Post-training Roles:

• Pre-training establishes zero/few-shot capabilities.
• RL tightly correlates with reward evolution ($\sim$0.8), improving downstream metrics linearly.
• Rollout length expands for reasoning, contracts for perception (entropy dynamics).
• PaCoRe externalizes corroborative "System 2" reasoning traces, boosting accuracy and interpretability.

7. Research Impact and Future Directions

STEP3-VL-10B presents a reproducible, efficient baseline for multimodal reasoning research. Its compact design, advanced synergy protocols, and robust post-training pipeline provide a blueprint for subsequent VL models prioritizing resource optimization without sacrificing competitive performance. The introduction of PaCoRe may have broader implications for ensemble, multi-agent, and meta-cognition architectures in VL systems. A plausible implication is the potential for further compression of “System 2” intelligence via design, not scale—a point of active investigation in foundational AI research (Huang et al., 14 Jan 2026).