Qwen-3 4B Backbone Overview

• Qwen-3 4B backbone is a 4 billion parameter, decoder-only transformer designed for efficient multilingual modeling and parameter-efficient fine-tuning.
• Its architecture features 36 layers with grouped query attention and rotary position embeddings to support extended context lengths.
• The model enables efficient adaptation via LoRA and tailored tokenizer enhancements, optimizing performance for resource-constrained environments.

The Qwen-3 4B backbone is a 4 billion parameter, decoder-only Transformer architecture designed as part of the Qwen LLM family, with specific adaptations for efficient parameter-efficient fine-tuning and multilingual capabilities. It serves as the foundational model for further developments, such as the Racka continual pretraining recipe for Hungarian language adaptation and related applications (Csibi et al., 3 Jan 2026). The backbone provides a highly optimized, well-regularized, and robust base suitable for large-scale language modeling tasks in resource-constrained high-performance computing environments.

1. Architectural Configuration

The Qwen-3 4B backbone comprises a stacked, decoder-only Transformer consisting of 36 layers, resulting in a total parameter count of approximately 4 billion—3.6 billion parameters in the backbone outside of token embeddings. Each Transformer layer has a hidden dimension of $d=2560$, and the feed-forward network (FFN) in each layer is two-staged with a total width of $4d=10240$.

The multi-head attention mechanism employs Grouped Query Attention (GQA): specifically, each layer uses 32 query heads and 8 key-value heads. Consequently, each query head has a dimension of $d/32=80$, and each key-value head has a dimension of $d/8=320$. Non-linearities within the sublayers are SwiGLU activations. All normalizations within the model use RMSNorm rather than the conventional LayerNorm, a choice that aligns with the pre-norm stabilization approach established within the broader Qwen series (Bai et al., 2023).

Positional information is encoded using rotary position embeddings (RoPE), enabling native support for contexts up to 32,000 tokens, with simple interpolation available for longer sequences.

2. Parameterization and Model Capacity

The parameter distribution is dominated by the Transformer stack, with 3.6 billion parameters exclusive of token embeddings. Embedding and output projection matrices account for the remainder, and these are untied for improved flexibility and performance. The backbone’s configuration is distinguished from examples in the original Qwen technical specifications, which detail 1.8B, 7B, and 14B configurations, by extending both depth (36 layers versus up to 40 in the largest Qwen) and width ($d=2560$ relative to reported values within the 1.8B–14B range) (Bai et al., 2023, Csibi et al., 3 Jan 2026).

3. Attention and Nonlinearity Details

Attention is implemented with Grouped Query Attention rather than the standard multi-head self-attention found in smaller Qwen variants. The use of GQA reduces the memory and compute overhead in the aggregation of queries and keys/values, supporting scalability in large-batch distributed environments. Each sublayer activation follows a SwiGLU (Swish-Gated Linear Unit) nonlinearity rather than the more common GELU or standard ReLU-type functions, and all normalizations utilize RMSNorm for stability and computational efficiency.

The attention computation within one layer comprises separate projection matrices for Q, K, and V, with additional output projection, resulting in an efficient communication pattern suitable for high-throughput hardware. The rotary position encoding mechanism used in Qwen-3 4B enables rapid access to relative positional information and supports efficient inference-time context extension.

4. Adaptations for Multilingual and Agglutinative Languages

In the Racka continual pretraining experiment, the Qwen-3 4B backbone was adapted for Hungarian through tokenizer extension and vocabulary augmentation. The original Qwen tokenizer was expanded by training a 32,000-token BPE tokenizer on Hungarian data and merging these subwords into the pre-existing vocabulary after pruning some non-Latin multi-byte tokens. The final vocabulary size was thus $$|\mathcal{V}^\textrm{new}|=|\mathcal{V}^\textrm{orig}|+32,000$$.

New embeddings were initialized with Vocabulary Initialization with Partial Inheritance (VIPI) by averaging the embeddings of the original-token decomposition of each new subword. Validation on Hungarian text reduced subword “fertility” from 3.13 to 1.66, while English and German fertility increased modestly (from 1.57 to 1.94, and 2.05 to 2.31, respectively) (Csibi et al., 3 Jan 2026).

5. Parameter-Efficient Adaptation: Low-Rank Adaptation (LoRA)

Adaptation of the Qwen-3 4B backbone for Hungarian in Racka was implemented using Low-Rank Adaptation (LoRA). All original Qwen-3 weights were frozen, and for every linear projection within attention and FFN blocks, LoRA adapters were interleaved, making only a small fraction of parameters trainable. Formally, each frozen weight matrix $$W\in\mathbb{R}^{d_{\textrm{out}}\times d_{\textrm{in}}}$$ was replaced at training time by $W' = W + \Delta W$, where $\Delta W = (\alpha/r) AB$ with $A\in\mathbb{R}^{d_{\textrm{out}}\times r}$, $B\in\mathbb{R}^{r\times d_{\textrm{in}}}$, $r=64$, and scaling factor $\alpha=128$. This design exposed approximately 0.52 billion parameters (12.5% of total) to training, while the majority of the model remained fixed.

A dropout of $0.1$ was applied on LoRA adapter outputs during training. This approach enabled practical large-scale continual pretraining using limited compute resources—distributed over A100 (40GB) clusters and optimized for low inter-node bandwidth (Csibi et al., 3 Jan 2026).

6. Training Protocols and Practical Considerations

Pretraining of the Qwen-3 4B backbone follows the autoregressive next-token prediction paradigm. In the Racka adaptation, continual pretraining leveraged 160B subword tokens drawn from a multilingual, code-inclusive mix: 44% Hungarian, 24% English, 21% German, and 11% source code. The data strategy sought to mitigate catastrophic forgetting of high-resource languages while facilitating adaptation to a morphologically rich, agglutinative language.

Training was conducted on the Komondor HPC cluster (16 nodes × 4 A100 GPUs each). Input sequences were “packed” into 4096-token causal contexts with 4D masks. An effective batch size of 512 sequences (physical batch size 2 × gradient accumulation 4 on 64 GPUs) was used. Optimization followed AdamW (weight decay 0.005), with learning rates of $10^{-4}$ (LoRA) and $5\times10^{-5}$ (non-LoRA), using linear warmup over the first 1% of steps and subsequent linear decay.

The framework stack included PyTorch 2.7.1, HuggingFace 4.52.3, Accelerate 1.7.0, and PEFT 0.15.2, with Distributed Data Parallel (DDP) for synchronization. Training spanned 326,357 steps, consuming approximately 287 hours (2.1 GPU-years) and emitted an estimated 1,290 kg CO$_2$-eq (Csibi et al., 3 Jan 2026).

7. Summary Table: Core Hyperparameters of Qwen-3 4B Backbone

Field	Value	Notes
Layers (L)	36	Decoder-only Transformer
Hidden size ($d$)	2560	All layers
FFN inner dimension	10240	Two-stage, $4d$
Attention heads (Q/KV)	32 Query, 8 Key-Value	Grouped Query Attention (GQA)
Activation	SwiGLU	Per-sublayer
Normalization	RMSNorm	Pre-norm
Positional encoding	RoPE	$32$K tokens, interpolation enabled
Total parameters	$\sim$4B (3.6B non-emb)	Untied embeddings
LoRA adaptation (Racka)	$r=64$, $\alpha=128$	$0.52$B trainable
Agglutinative tokenizer	+32,000 BPE (Hungarian)	VIPI embedding init

The Qwen-3 4B backbone presents a robust, scalable architectural template for both general-purpose and specialized language modeling. Its design choices are consistent with the larger Qwen model family, with enhancements for efficient multi-lingual adaptation, memory-intensive tasks, and sustainable operation across distributed academic infrastructure (Bai et al., 2023, Csibi et al., 3 Jan 2026).