DeepSeek-V3 Technical Report

Abstract: We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) LLM with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.

• The paper introduces a 671B-parameter MoE LLM featuring 37B active parameters per token with an innovative auxiliary-loss-free load balancing mechanism.
• It leverages Multi-head Latent Attention (MLA) and Multi-token Prediction (MTP) to reduce memory usage and densify the training signal.
• The report demonstrates state-of-the-art results on knowledge, math, and code benchmarks using FP8 mixed precision training and robust long-context modeling.

DeepSeek-V3: Architecture, Training, and Evaluation of a 671B-Parameter MoE LLM

Introduction and Motivation

DeepSeek-V3 represents a significant scaling and architectural advance in open-source LLMs, featuring a 671B-parameter Mixture-of-Experts (MoE) design with 37B active parameters per token. The model is trained on 14.8T tokens and incorporates several architectural and systems-level innovations to achieve high performance, cost efficiency, and robust long-context capabilities. The report details the model's architecture, training pipeline, infrastructure optimizations, and comprehensive evaluation, positioning DeepSeek-V3 as a leading open-source alternative competitive with closed-source models.

Figure 1: Benchmark performance of DeepSeek-V3 and its counterparts.

Model Architecture

Core Design: MLA and DeepSeekMoE

DeepSeek-V3 builds upon the Transformer backbone, integrating Multi-head Latent Attention (MLA) and DeepSeekMoE for efficient inference and economical training, respectively. MLA reduces the key-value (KV) cache size during inference by compressing keys and values into low-rank representations, which are then up-projected as needed. This design maintains comparable performance to standard Multi-Head Attention while significantly reducing memory and bandwidth requirements.

DeepSeekMoE employs fine-grained expert routing, with each MoE layer comprising 1 shared and 256 routed experts. For each token, 8 experts are activated, and routing is constrained to a maximum of 4 nodes to minimize communication overhead. Notably, DeepSeek-V3 introduces an auxiliary-loss-free load balancing strategy, replacing traditional auxiliary loss with a dynamic bias-based mechanism that adjusts expert selection to maintain balanced loads without degrading model performance.

Figure 2: Illustration of the basic architecture of DeepSeek-V3, highlighting MLA and DeepSeekMoE for efficient inference and training.

Multi-Token Prediction (MTP)

DeepSeek-V3 incorporates a Multi-Token Prediction (MTP) objective, extending the training signal by predicting multiple future tokens at each position. Unlike prior approaches that use parallel output heads, DeepSeek-V3's MTP implementation maintains the full causal chain for each prediction depth, using sequential modules that share embeddings and output heads with the main model. This densifies the training signal and enables speculative decoding for inference acceleration.

Figure 3: Illustration of the Multi-Token Prediction (MTP) implementation, maintaining the complete causal chain for each token at each depth.

Load Balancing and Expert Specialization

The auxiliary-loss-free load balancing strategy dynamically adjusts per-expert biases based on observed load, ensuring batch-wise balance without imposing strong sequence-wise constraints. This approach enables greater expert specialization, as evidenced by domain-specific load patterns, and consistently yields superior performance compared to auxiliary-loss-based methods.

Figure 4: Expert load comparison between auxiliary-loss-free and auxiliary-loss-based models, showing greater specialization in the former.

Training Infrastructure and Systems Optimizations

Distributed Training and DualPipe

DeepSeek-V3 is trained on a 2048 H800 GPU cluster, leveraging a custom HAI-LLM framework. The training pipeline employs 16-way pipeline parallelism (PP), 64-way expert parallelism (EP) across 8 nodes, and ZeRO-1 data parallelism. The DualPipe algorithm is introduced to maximize computation-communication overlap, reducing pipeline bubbles and hiding all-to-all and PP communication behind computation.

Figure 5: Overlapping strategy for forward and backward chunks, fully hiding all-to-all and PP communication.

Figure 6: Example DualPipe scheduling for 8 PP ranks and 20 micro-batches, illustrating overlapped computation and communication.

FP8 Mixed Precision Training

A fine-grained FP8 mixed precision framework is developed and validated at scale. Most GEMM operations are performed in FP8, with critical components (e.g., embeddings, output heads, normalization) retained in higher precision. Fine-grained quantization (tile-wise for activations, block-wise for weights) and high-precision accumulation (promotion to CUDA cores at 128-element intervals) are employed to mitigate quantization errors and maintain training stability. The framework also compresses cached activations and optimizer states to reduce memory and communication overhead.

Figure 7: Mixed precision framework with FP8 data format, showing the Linear operator.

Figure 8: (a) Fine-grained quantization to mitigate outlier-induced errors; (b) Improved FP8 GEMM precision via high-precision accumulation.

Loss curves demonstrate that FP8 training achieves a relative error below 0.25% compared to BF16, validating the approach for large-scale LLMs.

Data, Pre-Training, and Long-Context Extension

The pre-training corpus is constructed to enhance mathematical, programming, and multilingual content, with document packing and Fill-in-Middle (FIM) strategies to improve data integrity and modeling capabilities. The tokenizer uses byte-level BPE with a 128K vocabulary, optimized for multilingual compression.

Long-context capability is achieved via YaRN-based context extension, with two post-pretraining phases expanding the context window from 4K to 32K and then to 128K. DeepSeek-V3 maintains robust performance on the "Needle In A Haystack" (NIAH) benchmark across all context lengths.

Figure 9: NIAH evaluation results, demonstrating robust performance up to 128K context length.

Post-Training: Alignment and Distillation

Supervised fine-tuning (SFT) and reinforcement learning (RL) are applied post-pretraining. Reasoning data is distilled from DeepSeek-R1 models, incorporating verification and reflection patterns to enhance reasoning performance while controlling output style and length. RL employs Group Relative Policy Optimization (GRPO), using both rule-based and model-based reward models, and leverages self-rewarding via LLM-based voting for open-ended tasks.

Evaluation and Results

Comprehensive evaluation across knowledge, code, math, reasoning, and multilingual benchmarks demonstrates that DeepSeek-V3 outperforms all open-source models and is competitive with leading closed-source models such as GPT-4o and Claude-3.5-Sonnet. Notably, DeepSeek-V3 achieves:

• 88.5 on MMLU, 75.9 on MMLU-Pro, and 59.1 on GPQA, matching or surpassing closed-source models in educational and factual knowledge.
• State-of-the-art results on math (e.g., MATH-500, AIME, CNMO) and code (e.g., HumanEval, LiveCodeBench) benchmarks, with substantial margins over prior open-source models.
• Robust long-context performance and strong expert specialization, enabled by architectural and training innovations.

Implications and Future Directions

DeepSeek-V3 demonstrates that large-scale MoE LLMs can achieve high performance and efficiency through architectural, algorithmic, and systems co-design. The auxiliary-loss-free load balancing and MTP objectives are shown to be effective at scale, and the FP8 training framework sets a precedent for low-precision training in trillion-token regimes. The report also provides actionable hardware design recommendations, including support for fine-grained quantization, higher-precision accumulation, and communication offloading.

The model's open-source release, combined with its strong performance and cost efficiency (2.788M H800 GPU hours for full training), is expected to accelerate research and deployment of large-scale LLMs. Future work will focus on further architectural improvements, data scaling, deep reasoning capabilities, and more comprehensive evaluation methodologies.

Conclusion

DeepSeek-V3 establishes a new standard for open-source LLMs, combining a scalable MoE architecture, efficient training and inference, and robust alignment strategies. The innovations in load balancing, multi-token prediction, and FP8 training are validated at unprecedented scale, yielding a model that is both performant and accessible. The work provides a blueprint for future LLM development, emphasizing the importance of holistic optimization across model, algorithm, and hardware layers.