Mistral-7B-v0.3: Efficient Instruction Tuning

• Mistral-7B-v0.3 is a 7B-parameter decoder-only Transformer that enhances instruction-following with targeted domain adaptation and fine-tuning.
• The model employs a 32-layer architecture and L2 Adaptive Computation to dynamically skip unactivated layers, improving inference efficiency.
• Continued pre-training on domain-specific data combined with diverse instruction tuning yields significant gains on both defense-specific and general tasks.

Mistral-7B-v0.3 is a 7 billion-parameter, decoder-only Transformer LLM that serves as the instruction-tuned variant of the foundational Mistral-7B architecture. Developed to enhance instruction-following capabilities, v0.3 is prominently used as a baseline and adaptation target in both domain adaptation studies and investigations of inference-time computational redundancy. The model architecture, adaptation methodology, empirical characteristics, and evaluation results are detailed below, with particular emphasis on domain transfer and layer efficiency.

1. Model Architecture and Base Training

Mistral-7B-Instruct-v0.3 is instantiated as a 32-layer, decoder-only Transformer with each layer comprising 16 self-attention heads, a model (hidden) dimension $d_{\text{model}} \approx 8,192$, and a feed-forward inner dimension of approximately 32,768. The model supports a context window of up to 8,192 tokens. The original pre-training regime, conducted by Mistral AI, employed a next-token prediction objective: $$L_{\text{CE}}(\theta) = -\sum_{t=1}^T \sum_{v \in V} y_{t,v} \log p_\theta(v \mid x_{<t}),$$ where $y_{t,v}$ is the one-hot target for token $v$ at position $t$. This training utilized a blend of web-scraped corpora, code, and other public data sources. Model weights for v0.3 are further refined via supervised instruction fine-tuning (cross-entropy loss on a curated mixture of instruction and chat data), optimizing for adherence to human instructions (Shemiranifar, 20 May 2025, Rousseau et al., 7 Jul 2025).

2. Domain Adaptation: Continued Pre-training and Instruction Tuning

Comprehensive domain adaptation of Mistral-7B-Instruct-v0.3 has been investigated in the context of specialized low-footprint transfer. The adaptation protocol employs a two-phase regimen:

• Phase 1: Continued Pre-training (Poursuite du pré-entraînement, PPE)
  • Closed Regime: Exclusively AMIAD data (official defense sources, selected Wikipedia and internal documents), processed into $\approx 54,865$ segments ($\approx 48.6$M tokens after cleaning).
  • Open Regime: AMIAD plus Ouest-France archival news ($\approx 10,910$ articles, $11.5$M tokens).
  • Preprocessing includes segmentation by document or chunk, character length filtering, OCR for embedded text, and GPT-4o-mini extraction for acronyms/translations.
• Phase 2: Instruction Tuning (Affinage sur instructions, ASI)
  • Diverse synthetic and templated instructions are constructed from five sources, including AMIAD and Ouest-France-based examples, plus a filtered French subset of Tülu 3 SFT to promote generalist skills.
  • Instruction datasets range in size (see Table 1 below), and all use a three-part chat format (system, user, agent), with only agent tokens included in the loss calculation.

Table 1: Instruction Tuning Corpora

Source	#Instr.	#Tokens	Avg in/out (tokens)
Gen AMIAD	79,974	60.9M	577 → 93
Patrons AMIAD	2,228	138k	39 → 11
Long AMIAD	6,082	16.8M	272 → 2,008
Gen OF	32,912	28.0M	648 → 102
Tülu 3 Fr	5,891	10.0M	744 → 946

For fine-tuning, the Hugging Face TRL SFTTrainer and Deepspeed ZeRO 3 were utilized with hyperparameters: learning rate $2 \times 10^{-5}$, cosine scheduler, $5\%$ warmup, batch size $2$, gradient accumulation $64$, $3$ epochs, max grad norm $1.0$, weight decay $0.1$, packing enabled.

3. Layer Utilization and L2 Adaptive Computation (LAC)

Void-induced inefficiencies during inference were probed using L2 Adaptive Computation (LAC) (Shemiranifar, 20 May 2025). LAC detects activation “Voids” by monitoring the per-layer L2-norm changes of hidden states:

• Formally: For hidden state $\mathbf{h}^{(l)} \in \mathbb{R}^d$ at layer $l$,

$$\|\mathbf{h}^{(l)}\|_2 = \sqrt{\sum_{i=1}^d (h^{(l)}_i)^2}$$

The per-layer progress is $$\delta_t = \|\mathbf{h}^{(t)}\|_2 - \|\mathbf{h}^{(t-1)}\|_2$$.

• Dynamic halting threshold:

$$\lambda_t = \alpha \cdot (\max(\Delta_t) - \min(\Delta_t))$$

where $\Delta_t = \{\delta_1,\ldots,\delta_t\}$ and $\alpha \in (0,1]$ is a hyperparameter controlling aggressiveness.

• A layer is considered a Void (unactivated) on a token if $\delta_t < \lambda_t$, in which case its activation is zeroed for all subsequent layers.

Empirically, with $\alpha=0.8$, Mistral-7B-Instruct-v0.3 activates only $71$–$74\%$ of its layers on MMLU, GPQA Diamond, and BoolQ, skipping $\approx 26\%$ of layers per token. Notably, layers near the bottom and top persistently remain active, while some middle layers may drop below $50\%$ activation in certain tasks (Shemiranifar, 20 May 2025).

On GPQA Diamond, applying LAC (with $\sim74\%$ layer usage) increases accuracy from $13.88\%$ to $18.36\%$ without additional training, corroborating that a nontrivial fraction of computation can be masked—though practical FLOP or latency improvements await block-sparse or hardware-supported execution.

4. Empirical Results: Task Gains and Generalization

Adapted variants of Mistral-7B-Instruct-v0.3 were evaluated on both defense-domain and generalist tasks, using metrics such as accuracy and mean opinion score (MOS, [0–5], computed by GPT-4o-mini). Six adaptation runs were systematically compared—three in the closed regime (AMIAD-only PPE and instructions), three in the open regime (plus Ouest-France data and more varied instructions).

Defense-specific tasks (selected results):

Model	QCM (%)	Factual QA (%)	Acronym (%)	Résumé (MOS)	Titrage (MOS)
Mistral-7B base	57.5	5.6	3.9	3.24	3.27
Closed C1	65.4	9.2	3.9	3.93	3.81
Closed C2	64.4	6.8	8.6	3.86	3.76
Closed C3 (PPE)	62.2	10.3	10.9	2.10	2.06
Open O3 (domain+gen)	65.6	11.3	14.8	3.66	3.25

Combining continued pre-training with diversified instruction-tuning (especially run Open O3) achieves the highest factual QA and acronym accuracy. However, excessive emphasis on PPE over instruction variety can reduce generative writing MOS.

General-domain performance:

Model	QCM Gen (%)	MMLU En (%)	MMLU Fr (%)	IFEval En (%)	IFEval Fr (%)
Mistral-7B base	62.5	53.2	48.6	48.4	51.9
Closed C2	67.5 (+5)	53.3 (+0.1)	47.5 (–1.1)	54.5 (+6.1)	48.5 (–3.4)
Open O3	57.5 (–5)	48.4 (–5)	42.6 (–6)	50.3 (+1.9)	53.6 (+1.7)

ASI alone (no PPE, as in C2) preserves or marginally increases generalist accuracy. PPE alone tends to degrade general QCM/MMLU, but the combination with generalist instructions recovers instruction-following on IFEval.

5. Computational Cost and Carbon Footprint

Carbon footprint was estimated as $$C = P_{\text{GPU}} \times T_{\text{hours}} \times {EF}_{\text{CO}_2}$$ (in gCO₂e), where instruction generation (GPT-4.1-mini, Norway) for 16.9M tokens resulted in $\approx 1,890~$gCO₂e, and full adaptation (continued pre-training plus instruction-tuning on 8× NVIDIA H200, France) required an additional $930~$gCO₂e (PPE) and $385$–$623~$gCO₂e (ASI per run). In total, domain adaptation incurred only $\sim2.5~$kgCO₂e beyond instruction generation. ASI-only adaptation is especially efficient at $\sim0.4~$kgCO₂e per run, with strong task-specific improvements and minimal degradation of general capabilities (Rousseau et al., 7 Jul 2025).

6. Practical Considerations and Limitations

The two-step adaptation methodology demonstrates the feasibility of targeting high-value domains for relatively small LLMs using pragmatic energy budgets. Selecting training corpora for instruction tuning is critical: exclusive domain adaptation without generalist examples reduces broad task performance, while balanced mixtures recover or improve generalist skills. From an inference perspective, LAC shows that a static Transformer like Mistral-7B-Instruct-v0.3 need not use all layers per token—substantial portions may be skipped without retraining, even improving task metrics on some benchmarks. However, actual computational savings in FLOPs require hardware support for dynamic halting or block-sparse inference.

7. Summary and Impact

Mistral-7B-Instruct-v0.3 provides a robust, extensible reference for instruction tuning and domain specialization within the 7B parameter class. Empirical evidence demonstrates that targeted continued pre-training and instruction adaptation meaningfully improve task performance for defense-specific (and plausibly other) domains, while judicious mixing of instruction sources safeguards generalization. The application of L2 Adaptive Computation further establishes that not all layers are needed for all tokens, indicating opportunities for more efficient inference architectures. These factors collectively demonstrate the viability of scalable, low-footprint domain adaptation of LLMs such as Mistral-7B-Instruct-v0.3, aligning state-of-the-art instruction-following with domain-specific and efficiency objectives (Shemiranifar, 20 May 2025, Rousseau et al., 7 Jul 2025).