Qwen-3-8B: Clarifying a Nonexistent Variant

• Qwen-3-8B is a misinterpreted label, as it is not included in the official Qwen series which only comprises Qwen-1.8B, Qwen-7B, and Qwen-14B models.
• The Qwen series employs transformer architecture innovations such as untied input/output embeddings, RoPE in FP32, and SwiGLU activation to enhance scaling and long-context performance.
• Qwen-7B serves as the mid-scale representative, demonstrating competitive benchmark performance and providing key insights into training paradigms and model efficiency.

Qwen-3-8B is not a described or released variant within the Qwen LLM series. The Qwen Technical Report details three primary base models: Qwen-1.8B, Qwen-7B, and Qwen-14B, with no mention of a Qwen-3-8B or any model at the 3.8B parameter scale. The following sections summarize the closest published models, with Qwen-7B serving as the representative for mid-scale architecture and performance characteristics (Bai et al., 2023).

1. Model Series Overview

The Qwen LLM suite comprises pretrained transformers of various capacities. Qwen base models are trained on language modeling objectives, while derivative models—such as Qwen-Chat, Code-Qwen, and Math-Qwen-Chat—target specialized applications including human-aligned dialog, coding, and mathematics. All Qwen models are based on the transformer architecture, integrating specific design innovations to maximize performance at their respective scales.

2. Architecture and Innovations

The Qwen-7B model, which provides the architecture most analogous to hypothetical mid-sized models, consists of:

• 32 transformer layers
• Hidden size of 4096
• 32 attention heads (with $d_k = 128$)
• Feed-forward network (FFN) inner dimension is $(8/3) \cdot 4096 \approx 10{,}922$

Architectural distinctions relative to vanilla GPT and LLaMA include:

• Untied input/output embeddings to enhance expressivity
• RoPE (rotary positional encoding) with parameters stored in FP32, improving long-range stability
• QKV bias terms for superior length extrapolation
• Pre-normalization using RMSNorm instead of LayerNorm
• SwiGLU activation with an FFN dimension of $8/3 \cdot d$, compared to the $4 \cdot d$ ratio common in other models

Self-attention is computed using the standard formula:

$$\mathrm{Attention}(Q, K, V) = \operatorname{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right) V$$

3. Model Scaling and Parameters

The Qwen series follows a discrete set of model sizes, with no 3.8B-parameter instance documented. Qwen-7B contains approximately 7 billion parameters, distributed as follows:

Component	Parameter Count (approx)
Token embeddings	0.3B
Output projection	0.3B
Attention projection/biases	1.0B
Feed-forward weights	3.4B
RMSNorm & miscellaneous	2.0B

No specific scaling law or architecture is supplied for an intermediate model at the 3.8B parameter scale. The paper specifies a sequence of 1.8B, 7B, and 14B models, reflecting compute and data optima (Bai et al., 2023).

4. Pretraining Data and Tokenization

Qwen models are pretrained on a diverse corpus totaling approximately 3 trillion tokens. The data sources encompass web documents, books, code, encyclopedias, and multilingual content, with a primary focus on English and Chinese. The preprocessing pipeline includes:

• Exact-match normalization and MinHash-LSH fuzzy hashing for deduplication
• Filtering by rule-based heuristics, automated quality-scoring, and manual data inspection
• Instruction-data upsampling with n-gram overlap filtering to support downstream instruction following

Tokenization employs a BPE vocabulary (tiktoken cl100k) augmented for Chinese granularity (character/word pieces, split digits), yielding a final vocabulary of approximately 152,000 tokens.

5. Training Process

All Qwen base models are trained using standard autoregressive cross-entropy, where the objective is:

$L = -\sum_t \log P_\theta(x_t \mid x_{<t})$

Key training parameters for Qwen-7B include:

• Context length: 2048 tokens
• Optimizer: AdamW with $$(\beta_1 = 0.9, \beta_2 = 0.95, \epsilon = 10^{-8})$$
• Learning rate: peak $3 \times 10^{-4}$, minimum is 10% of peak, cosine decay schedule
• Batch size: 4 million tokens per step, mixed-precision BFloat16, FlashAttention enabled
• Total tokens: 2.4 trillion (7B model), no gradient accumulation beyond standard shuffling

6. Benchmark Performance

Qwen-7B demonstrates competitive or superior performance versus open-source peers across major benchmarks:

Benchmark	Qwen-7B	Baichuan-7B	LLaMA2-7B	Code LLaMA-7B
MMLU (5-shot)	58.2%	54.7%	46.8%	N/A
C-Eval (5-shot)	63.5%	56.3%	32.5%	N/A
GSM8K (8-shot)	51.7%	N/A	16.7%	N/A
BBH (3-shot)	45.0%	N/A	38.2%	N/A
HumanEval (0-shot)	29.9%	N/A	N/A	33.5%

These results indicate substantial headroom over LLaMA2-7B and Baichuan-7B for both language understanding and code synthesis tasks (Bai et al., 2023).

7. Ablations and Analysis

Ablation studies and architectural analyses in the Qwen Technical Report highlight:

• Untied input/output embeddings confer a 0.5–1.0% gain in reasoning benchmarks, with an added $\approx 0.6$B parameter cost.
• RoPE in FP32 gives consistent improvements to long-context extrapolation versus FP16.
• QKV bias terms enable length extrapolation beyond 8,000 tokens without retraining.
• SwiGLU activation with larger FFN dimension achieves better early scaling than GeLU + $4d$ at comparable compute.
• Long-context inference enhancements (NTK-aware RoPE interpolation, LogN-scaling, windowed attention) maintain low perplexity up to 16,000 input tokens.

No evidence, ablation, or architectural outline exists for a hypothetical Qwen-3-8B variant.

---

All factual content is strictly drawn from the Qwen Technical Report (Bai et al., 2023), and there is no officially released or described Qwen-3-8B model therein. Qwen-7B constitutes the principal reference for a mid-scale Qwen base LLM.