LongCat-Flash-Lite: Scalable MoE & Embeddings

• LongCat-Flash-Lite is a large-scale language model that combines extensive N-gram embedding tables with MoE sparsity to break the efficiency–performance trade-off.
• It employs a transformer backbone with 14 shortcut layers and optimized CUDA kernels, achieving significant throughput speedups and reduced latency.
• Nearly 46% of its 68.5B parameters are dedicated to embedding layers, leading to superior agentic and coding performance compared to traditional MoE baselines.

LongCat-Flash-Lite is a large-scale LLM architecture that combines large N-gram embedding tables with Mixture-of-Experts (MoE) sparsity, introducing embedding scaling as an orthogonal dimension for parameter expansion. It consists of 68.5 billion total parameters, with only approximately 2.9–4.5 billion parameters activated per token during inference due to sparsity. Notably, over 30 billion parameters—roughly 46% of the total—are allocated to highly-parameterized embedding layers rather than MoE experts. This approach is empirically shown to surpass parameter-equivalent MoE baselines and demonstrate superior performance in agentic and coding domains, primarily by breaking through the efficiency–performance trade-off imposed by expert scaling limits (Liu et al., 29 Jan 2026).

1. Model Structure and Parameterization

LongCat-Flash-Lite employs a deep transformer backbone equipped with 14 shortcut layers, each containing an MoE block comprising 256 non-zero experts and 128 "zero" experts (placeholders with no learned parameters). Each token is routed to $k=12$ experts per MoE block. Embedding scaling is central: a base vocabulary of size $V_0 = 128\,\mathrm{K}$ is represented by an embedding matrix of hidden dimension $D=8\,\mathrm{K}$ for illustration. This is expanded with N-gram embedding tables up to order $N$, yielding total embedding parameters: $$P_e = V_0 D + \sum_{n=2}^N \sum_{k=1}^K V_{n,k} \frac{D}{(N-1)K}$$ where $K$ is the number of hash-based sub-tables per N-gram order and $V_{n,k}$ is the vocabulary size for the $k^\mathrm{th}$ sub-table of order $n$. In the final configuration, $P_e \approx 31.4\,\mathrm{B}$, representing nearly half of the total parameter count. Per-token active parameters remain low because only the embeddings and a small subset of MoE experts are used for each forward pass.

Under a parameter-equivalent MoE-only baseline ("Vanilla"), reallocating the embedding parameters into the MoE increases the number of experts from 256 to approximately 750 per layer, but this results in diminishing returns and increased system overheads. Embedding lookups, by contrast, scale as $O(1)$ and mitigate the need for additional inter-GPU communication.

2. Theoretical Foundations and Scaling Regimes

A key theoretical construct is the effective sparsity ratio, $r = P/P_\mathrm{act}$. Empirical observations reveal the following regime-dependent behaviors in training loss $\ell$:

• Low $r$ ($r < r_s$): MoE scaling (adding experts) is most effective, with loss scaling approximately $\ell_{\textrm{MoE}}(r) \approx a - b \log r$.
• Intermediate $r \approx r_s$: Curves for MoE scaling and embedding scaling intersect, delineating an optimal allocation point.
• High $r > r_s$: Further MoE expert scaling hits diminishing returns; embedding scaling achieves a lower loss.

The architecture's width $W$ and depth $L$ interact with the N-gram embeddings, with deeper models attenuating embedding signals and wider models amplifying them. To remain on the favorable branch of the observed U-shaped curve in $\ell(P_e/P)$, embedding parameters should not exceed 50% of the total ($P_e \leq 0.5P$). N-gram embeddings are introduced only after the MoE expert count exceeds its empirical "sweet spot." Embedding vocabularies $V_n$ are chosen to avoid multiples of $V_0$ to reduce hash collisions.

3. System-Level Optimizations and Inference Acceleration

LongCat-Flash-Lite leverages a device-resident N-gram cache, akin to a key–value (KV) cache, with specialized CUDA kernels that fuse operations such as AllReduce, ResidualAdd, LayerNorm, as well as kernels for quantized activation folding and expert selection. This fusion, combined with an optimized attention-combine kernel, reduces critical path latency by approximately 50%.

The model supports Programmatic Dependent Launch (PDL), enabling dependent kernels to be launched early, thus increasing streaming multiprocessor (SM) utilization by overlapping dependent operations. For decoding, a draft–verify–commit speculative decoding scheme is implemented:

def speculative_decode(input, T): draft_tokens = DraftModel.generate(input, T) verify_probs = MainModel.score(input + draft_tokens) accepted = RejectLowConfidence(draft_tokens, verify_probs) return accepted

This speculative pipeline, combined with expert parallelism and batch overlap, boosts effective batch size and throughput. On an 8×H800-80G infrastructure, the model achieves a 2.3× throughput speedup compared to baselines with identical input/output sequence lengths.

4. Training Regime and Experimental Protocol

LongCat-Flash-Lite is pre-trained on 11 trillion tokens at sequence length 8,000, followed by mid-training on 1.5 trillion tokens with a sequence length of 128,000, supported by the YaRN extension to reach up to 256,000 tokens. Supervised finetuning uses curated SFT data. Training occurs on hundreds of A100/H800 GPUs, leveraging ZeRO-3 parallelism (for MoE experts) and custom NCCL-based sharding.

For low-scale ablations, $P_\textrm{act}$ is varied by sweeping width $W$ with fixed depth $L=10$ and, for depth studies, fixing $W$ and varying $L$ across $\{10, 20, 40\}$. Embedding hyperparameters sweep $N \in [2,5]$, $K \in [1,4]$, and hash-based vocabularies are selected at $\sim30\times V_0$ with non-aligned sizing to reduce collisions.

5. Empirical Results and Comparative Benchmarks

LongCat-Flash-Lite is benchmarked against parameter-equivalent MoE models and external contemporaries (Kimi-Linear-48B, Qwen3-Next-80B, Gemini-2.5 Flash-Lite) on a suite of agentic, coding, and general-language tasks.

Table 1. Base Model Performance (Zero-Shot Accuracy)

Benchmark	Vanilla MoE (68.5B, 3Bₐ)	LongCat-Flash-Lite
MMLU	64.81	67.21
CEval	64.09	—

Table 2. Agentic & Coding Benchmarking

Benchmark	Kimi48	Qwen80	Gemini	LCFL (LongCat-Flash-Lite)
τ²-Bench Telecom (avg@8)	15.7	13.2*	21.9	72.8
SWE-Bench (acc)	32.8	37.6	41.3*	54.4
MMLU (acc)	79.9	89.3*	84.7	85.5
AIME24 (avg@32)	70.5	81.4*	63.3	72.2

*Note: Some Qwen80/Gemini results marked with an asterisk refer to reported best upstream numbers.

LongCat-Flash-Lite demonstrates substantial improvements in agentic tool use, coding, and reasoning tasks, with particularly strong zero-shot and agentic performance. It achieves both lower active-parameter I/O at decoding and competitive wall-clock throughput.

6. Broader Implications, Limitations, and Prospective Directions

Embedding scaling in LongCat-Flash-Lite enables an expanded Pareto frontier in parameter efficiency beyond the MoE "sweet spot" ($r = P/P_\textrm{act}$), yielding improved performance for a given activation budget. N-gram embeddings furnish richer local contextualization to the model, abetting zero-shot generalization and agentic task completion while containing compute and interconnect demands at inference time.

However, allocating large Pe imposes elevated GPU memory requirements. Hash collisions and embedding initialization require precise management, particularly as vocabularies scale. Deep architectures ($L \gg 40$) may attenuate embedding impact, constraining some scaling benefits.

Future directions articulated include: deploying N-gram branches as standalone draft models, instituting early rejection via embedding confidence for speculative decoding, exploring per-layer N-gram embeddings with dynamic allocation, and extending embedding scaling to multi-modal and retrieval-augmented setups.

LongCat-Flash-Lite establishes that large embedding allocations, paired with MoE sparsity and system-level acceleration, yield performance and efficiency competitive with or exceeding specialized MoE and dense models in the 48–80B parameter regime for both language and coding tasks, substantiated by extensive empirical validation (Liu et al., 29 Jan 2026).