TinyLlama 1.1B Model

• TinyLlama is a compact transformer-based language model family featuring rigorous pretraining, architectural refinements, and efficient deployment strategies.
• It employs a decoder-only architecture with 1.1B parameters, incorporating innovations like Rotary Positional Embeddings, RMSNorm, and SwiGLU activation.
• Quantization techniques using INT8 and INT4 deliver reduced memory usage and energy consumption, making TinyLlama ideal for edge computing applications.

TinyLlama is a family of compact, open-source transformer-based LLMs, with the flagship variant comprising 1.1 billion parameters. Designed to combine rigorous pretraining, architectural refinements, and efficient deployment strategies, TinyLlama targets the intersection of on-device inference, competitive accuracy, and minimal resource footprint. The model’s open-source releases, instructional tuning variants, and edge-centric profiling have positioned it as a canonical reference for research on small LLMs suitable for edge computing, privacy-preserving applications, and resource-constrained environments (Zhang et al., 2024).

1. Architectural Foundations

The core TinyLlama-1.1B model employs a decoder-only transformer architecture derived from Llama 2. It comprises 22 transformer blocks, each with a hidden size of 2,048 and 16 attention heads (head dimension 128). The feed-forward (MLP) dimension is 5,632. The model's vocabulary size is 32,000, utilizing Llama 2’s Byte-Pair Encoding tokenizer. The design includes several architectural adaptations:

• Rotary Positional Embeddings (RoPE): For efficient and injective position encoding, improving extrapolation and token context management.
• RMSNorm Pre-normalization: Replacing conventional LayerNorm for superior training stability.
• SwiGLU Activation: Utilized in the feed-forward layers instead of the standard ReLU, supporting improved gradient flow.
• Grouped-Query Attention: Implements 32 query heads with 4 shared key-value groups, optimizing the Key/Value memory overhead without loss of representational power (Zhang et al., 2024).

Parameter accounting adheres to standard transformer decompositions:

• Self-attention block: $4d^2$ parameters per layer (Q, K, V, O projections).
• Feed-forward block: $2d \, d_{ff}$ per layer.
• Embeddings: $|V| \times d$.
• Total parameter count for TinyLlama-1.1B: $\sim$1.1 billion.

2. Pretraining Corpus and Optimization

TinyLlama-1.1B is pretrained on approximately 1 trillion tokens for nearly 3 epochs. The corpus comprises SlimPajama (627B tokens of filtered RedPajama text) and 250B tokens from StarCoderData (code and natural-language pairs). After deduplication, about 950B unique tokens remain, partitioned at a natural:code ratio of 7:3. Notably, removal of duplicated GitHub content was performed to ensure clean web/code balance.

Pretraining is executed in autoregressive fashion with cross-entropy loss:

$\mathcal{L} = -\sum_{t=1}^T \log p(x_t | x_{<t}),$

using AdamW (β₁=0.9, β₂=0.95, weight decay=0.1), a cosine decay learning rate schedule (4.0e–4 to 4.0e–5), and full sharding across multi-GPU nodes via FSDP. System-level optimizations leverage FlashAttention 2, fused normalization, and batch sizes up to 2 million tokens per step. A significant downstream quality jump was observed after a bug fix (removal of excess EOS tokens) at 2.3T tokens into training (Zhang et al., 2024).

3. Quantization and Edge Profiling

TinyLlama is systematically profiled for edge deployment with aggressive quantization strategies (Pinnock et al., 6 Jun 2025). The primary schemes are:

• INT8 Quantization: 8-bit weights and activations.
• INT4 Quantization: 4-bit weights (symmetric per-channel for weights, asymmetric per-tensor for activations).

Quantization is post-training; scale and zero-point values are calibrated on a held-out corpus via min/max statistics. No Quantization-Aware Training is performed, yet measured accuracy loss remains within 2–5% (INT4), and below 2% (INT8) compared to FP16 baselines (Pinnock et al., 6 Jun 2025, Wang et al., 2024).

Analytical models under the EdgeProfiler framework estimate compute, memory, latency, and energy for TinyLlama deployed on edge hardware. Key equations include:

• FLOPs per token:

$$\text{FLOPs}_{\text{token}} = L (6H^2 + 4HS + 4HI + 4IH + 9H)$$

• Memory usage:

$$M = P\cdot B + S\cdot H\cdot B + 2L\cdot S\cdot H\cdot B$$

• Energy per token:

$$E = \text{FLOPs}_{\text{token}} \cdot e_{\text{flop}} + M \cdot e_{\text{byte}}$$

where $B$ is bytes per value (2 for FP16, 1 for INT8, 0.5 for INT4), $P$ is total parameter count, $S$ the sequence length, and $H$ the hidden size.

Empirical evaluation on Raspberry Pi 4/5 and Jetson Orin Nano Super yields:

• INT8: ≈28% memory reduction, ≈1.86× faster, ≈35–50% lower energy per token.
• INT4: ≈43% memory reduction, ≈2.45× faster, ≈50–60% lower energy per token.

These results underscore that 4-bit quantized TinyLlama models can be deployed with sub-2 GB memory footprints and 2–3× speedups without substantial accuracy penalties (Pinnock et al., 6 Jun 2025).

4. Downstream Task Performance

TinyLlama-1.1B achieves strong results on general and specialized benchmarks when compared to similarly sized models.

Commonsense Reasoning (LM Evaluation Harness)

Model	HellaSwag	OBQA	WinoGrande	ARC-C	ARC-E	BoolQ	PIQA	Avg
TinyLlama	59.20	36.00	59.12	30.10	55.25	57.83	73.29	52.99

Problem-Solving (InstructEval)

Model	MMLU	BBH	HumanEval	DROP	Avg
TinyLlama	25.34	29.65	9.15	15.34	19.87

(Zhang et al., 2024)

Mobile Health Event Prediction

In real-world health-monitoring tasks using PMData (Fitbit logs, user self-reports), TinyLlama-1.1B (4-bit quantized) achieves:

• 4.31 GB RAM footprint on iPhone 15 Pro Max.
• 0.48 s Time-to-First-Token and ~2.14 s total end-to-end latency.
• Top SLM results: MAE=0.4214 for stress, MAE=1.5652 for readiness, and competitive values for fatigue and sleep quality.

TinyLlama matches or outperforms other SLMs in these regression tasks while using markedly less memory and with sub-second latency, indicating suitability for privacy-preserving, always-on health feedback (Wang et al., 2024).

Function Calling and Agentic Tasks

The instruction-tuned TinyLlama-1.1B-32k-Inst achieves 19.73% overall accuracy on the Berkeley Function Calling Leaderboard (BFCL), underperforming Qwen3-0.6B and xLAM-2-1B-fc-r baselines (45.76% and 53.97%, respectively). Notably, its accuracy on multi-turn tasks is 0% without further SFT, PEFT, or reinforcement-based optimization, indicating that prompt-only instantiations are insufficient for robust agentic applications (Haque et al., 27 Nov 2025).

5. Deployment Strategies: Edge and Embedded Systems

TinyLlama’s deployment on edge devices is enabled by aggressive quantization and tailored inference pipelines. The profiling with EdgeProfiler yields practical guidance for low-latency deployments:

• Quantization: INT8 is generally recommended for microcontrollers due to negligible accuracy loss and 40% energy savings, while INT4 is reserved for cases mandating the most stringent memory and compute footprints (Pinnock et al., 6 Jun 2025).
• Latency Dominated by Storage I/O: In highly optimized scenarios, storage I/O (not compute) dictates end-to-end runtime. The use of high-bandwidth DRAM or NVMe, and double buffering (e.g., PCIe on Jetson platforms), is advised.
• On-device Design: Minimal off-chip memory usage is unlocked via quantization, fusing memory transfers, and overlapping computation–I/O pipelines (Pinnock et al., 6 Jun 2025).

Distributed inference on ultra-low-power MCUs (TinyLlama-42M) demonstrates super-linear speedup (26.1×) and a corresponding 27.2× EDP improvement compared to single-MCU execution, confirming scalable deployment even at extreme edge with modest model sizes (42M params) (Bochem et al., 2024).

6. Training, Fine-tuning, and Personalization

TinyLlama’s open-source checkpoints and training scripts facilitate community-driven fine-tuning. While the base (1.1B) model is pretrained and, in some versions, instruction-tuned (e.g., with UltraChat dialog data), studies show that zero-shot personalization for health tasks is achieved via input-prompt engineering rather than gradient-based adaptation (Wang et al., 2024).

Parameter-efficient methods (LoRA, QLoRA) and preference alignment strategies (DPO, RLHF) have not yet been applied to TinyLlama in several major evaluation studies. Empirically, further fine-tuning on domain or agentic tasks is necessary to address accuracy shortfalls observed in function-calling and multi-turn reasoning (Haque et al., 27 Nov 2025). A plausible implication is that model improvements for agentic edge applications will require hybrid optimization pipelines beyond prompt-based deployment.

7. Open-Source Ecosystem and Broader Impact

TinyLlama’s full release includes pretraining/fine-tuning code, data processing workflows, HuggingFace model compatibility, and multi-node inference scripts. The model demonstrates higher throughput (24,000 tok/s per A100-40GB GPU) and lower training cost per token relative to Pythia-1.0B and MPT-1.3B. The repository and checkpoints are at https://github.com/jzhang38/TinyLlama (Zhang et al., 2024).

TinyLlama’s impact is visible across efficiency-driven research agendas, real-time on-device analytics (notably in health and IoT sensor platforms), and as a baseline for ablation and quantization studies. Its design tradeoffs have established empirical limits for small model accuracy in resource-limited function-calling and multi-step agentic reasoning, catalyzing new work in parameter-efficient adaptation and edge-centric ML system design.

---

References:

• TinyLlama: An Open-Source Small LLM (Zhang et al., 2024)
• EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model (Pinnock et al., 6 Jun 2025)
• Efficient and Personalized Mobile Health Event Prediction via Small LLMs (Wang et al., 2024)
• Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs (Bochem et al., 2024)
• TinyLLM: Evaluation and Optimization of Small LLMs for Agentic Tasks on Edge Devices (Haque et al., 27 Nov 2025)