Ministral 3: Efficient Dense Language Models

• Ministral 3 is a suite of dense, decoder-only transformer models featuring long-context (256K tokens) support and three sizes (3B, 8B, 14B) designed for compute- and memory-constrained applications.
• The models employ a cascade distillation approach with iterative layer, hidden-dimension, and FFN pruning to achieve substantial compression while maintaining competitive performance.
• Each model variant—base, instruct, and reasoning—utilizes tailored fine-tuning strategies including supervised tuning, logit distillation, and chain-of-thought optimization for robust, multimodal, and task-specific applications.

Ministral 3 is a family of parameter-efficient dense LLMs introduced for compute- and memory-constrained applications, with support for long-context comprehension and multimodal (text+image) tasks. The series comprises three model sizes—3B, 8B, and 14B parameters—each delivered in three functionally distinct variants: a pretrained base, an instruction-finetuned model, and a reasoning-optimized version. Ministral 3 models are released under the Apache 2.0 license, enabling unrestricted commercial and research usage (Liu et al., 13 Jan 2026).

1. Model Family Composition and Architectural Design

The Ministral 3 architecture is a dense, decoder-only transformer, derived via Cascade Distillation from the Mistral Small 3.1 (24B) foundation. All sizes share a uniform context window of 256,000 tokens, with grouped-query attention and FlashAttention for efficient throughput. Model configuration details:

Model	Layers	Hidden Dim	FFN Dim	Q/KV Heads	Context
Ministral 3 14B	40	5,120	16,384	32Q/8KV	256K
Ministral 3 8B	34	4,096	14,336	32Q/8KV	256K
Ministral 3 3B	26	3,072	9,216	32Q/8KV	256K

Each model is issued as three variants:

• Base: Dense transformer, distilled from MS3.1, trained on 1–3 T tokens (mixed text and multimodal).
• Instruct: Supervised and preference-aligned via SFT and ODPO, with logit distillation and fp8 quantization.
• Reasoning: SFT with chain-of-thought traces, followed by Group Relative Policy Optimization (GRPO) on STEM (math/code/visual) and general domains, then ODPO refinement.

Pruning and distillation are iteratively applied to produce each size, preserving maximal capability after parameter reduction (Liu et al., 13 Jan 2026).

2. Cascade Distillation: Compression and Training Methodology

Cascade Distillation is the core technical recipe underpinning the efficiency of Ministral 3. Compression proceeds through:

• Layer Pruning: $$\mathrm{score}_\ell = \mathbb{E}[\|{\rm out}_\ell\| / \|{\rm in}_\ell\|]$$; retain highest-scoring layers.
• Hidden-Dimension Pruning: PCA on attention and FFN, then basis rotation and truncation.
• FFN Pruning: Retain neurons in SwiGLU blocks via $\mathbb{E}|\mathrm{SiLU}(W_1x)\cdot W_3x|$ scores.

Distillation is logit-only (forward KL divergence with temperature scaling), which empirically outperforms mixed cross-entropy objectives. The approach for context window extension is a two-stage curriculum: begin with short (16K), then extend to 256K tokens using YaRN and position-wise softmax temperature scaling.

This staged pruning/distillation process enables the production of models substantially smaller than the teacher (e.g., 14B is $\sim$42% smaller than 24B Mistral Small 3.1) with competitive language and multimodal performance (Liu et al., 13 Jan 2026).

3. Instruction-Tuning and Alignment Approaches

For the Instruct variants, fine-tuning proceeds from the Base checkpoint in two phases:

• Supervised Fine-Tuning (SFT): Trained on heterogeneous, high-quality instruction datasets, including text and image.
• Logit Distillation: Student aligns logits with a larger Mistral Medium 3 teacher.
• Online Direct Preference Optimization (ODPO): Uses pairwise reward modeling and $\beta$-rescaling; tool execution is enabled for more realistic completion signals.

The Reasoning models further utilize chain-of-thought (CoT) supervision and Group Relative Policy Optimization (GRPO), a policy-gradient method that incorporates both STEM-domain RL and broader generalization via LLM-judge rubrics. ODPO is again applied as a final post-alignment step, discarding “thinking chunks” in scoring.

Distinctive in Ministral 3’s alignment pipeline is the explicit incorporation of adversarial and noisy prompts, as well as counter-factual consistency checks, to bolster real-world invariance (Bogavelli et al., 9 Jan 2026, Liu et al., 13 Jan 2026).

4. Robustness Evaluation and Comparative Performance

Ministral 3 8B exhibits state-of-the-art robustness to prompt perturbations, as assessed on a comprehensive enterprise benchmark suite spanning five major perturbation classes: general (e.g., whitespace, spelling, paraphrase), positional, format (JSON, YAML, XML, HTML), multilingual, and cross-lingual.

Robustness is measured via:

$$R = 1 - \left(\Delta_{\mathrm{Content}} \times \Delta_{\mathrm{Quality}}\right)$$

where $\Delta_{\mathrm{Content}}$ is the fraction of examples showing a content change (human-judged similarity $<$ 3/3) and $\Delta_{\mathrm{Quality}}$ is the fraction of content-shifted examples whose task-specific metric is altered.

Ministral 3 8B achieves:

• Overall $R = 89.40 \pm 0.73$
• General: $92.49$
• Positional: $78.92$
• Format: $89.45$
• Multilingual: $89.12$
• Cross-lingual: $87.66$

This score exceeds GPT OSS 120B (+2.11 points), is within 1.61 points of GPT 5.2 (Large), and outperforms the scale-matched Llama 3.1 8B by +19.88 points. Its positional robustness, though the weakest among categories, remains superior to many larger models. In multilingual and cross-lingual regimes, Ministral 3 8B exhibits the lowest quality delta (ranging 26.15–31.31 across eight languages), offering stable deployment for internationally varied input.

Perturbation-specific failures are still observed (notably in YAML format: $R=58.03$), but performance on JSON ($R=92.03$) and XML ($R=90.60$) confirms wide applicability in enterprise data pipelines (Bogavelli et al., 9 Jan 2026).

5. Multimodal Capabilities and Benchmarks

All Ministral 3 models natively process both text and image inputs. Visual features are encoded via a 410M-parameter ViT module (borrowed from Mistral Small 3.1), projected into the transformer’s hidden dimension. Training leverages capped and interleaved datasets (text, captioning, VQA, multimodal CoT).

Competitive results are reported on:

• MMMU (Multimodal Machine Understanding): Ministral 3 14B Base 59.9 vs. teacher’s 59.1
• MathVista (visual math): Ministral 3 14B Base 43.6 vs. teacher’s 51.3

These outcomes indicate minimal loss from parameter reduction and demonstrate that the compression method effectively transfers multimodal competencies (Liu et al., 13 Jan 2026).

6. Memory, Computation, and Deployment Characteristics

Ministral 3 is optimized for real-world resource constraints:

• 3B model: Fits on a single 16 GB GPU or edge device.
• 8B model: Requires approximately 24–28 GB GPU (fp16 inference).
• 14B model: Operable within 40–44 GB GPU memory.

Grouped-query attention with FlashAttention yields 2× speedup over standard multi-head attention. Context extension has sub-5% computational overhead.

Compared to contemporary open LLMs (Qwen 3, Gemma 3), Ministral 3 demonstrates similar or superior results on MMLU-Redux, TriviaQA, MATH, and other established academic benchmarks, at reduced training cost and steady scaling across size tiers.

Sample usage with HuggingFace Transformers is demonstrated for both text-only and vision+chat modalities, with context windows up to 256K tokens (128K for the reasoning variant) (Liu et al., 13 Jan 2026).

7. Licensing, Use Cases, and Practical Deployment

All models are released under Apache 2.0, with commercial and research freedom, patent grant, and no copyleft restrictions. Recommended utilization scenarios include:

• On-device inference (using 3B for consumer platforms).
• Edge AI in robotics and appliances (leveraging reasoning+vision).
• Budget-optimized cloud/vps deployment (8B as tradeoff point).
• Long-document processing and analysis (full 256K context).
• Multimodal agent architectures (language, vision, and structured outputs).

For enterprise adoption, the robustness profile of Ministral 3 8B provides operational stability across linguistic, formatting, and prompt-structure variances, with cost-performance advantages over much larger LLMs. Regular perturbation-informed benchmarking is advised, and retraining for positional robustness may further optimize workflow reliability (Bogavelli et al., 9 Jan 2026, Liu et al., 13 Jan 2026).