Byte-Level Supervised Fine-Tuning

• Byte-level supervised fine-tuning is a method that trains neural networks to directly map raw byte sequences to task-specific outputs, eliminating traditional tokenization.
• It leverages specialized protocols such as adapter modules and head adaptation to effectively handle fuzz testing and language model distillation.
• Empirical results demonstrate significant gains in mutation efficiency and crash detection with only a modest drop in downstream language task accuracy.

Byte-level supervised fine-tuning is a methodology for adapting large-scale neural models to operate directly on raw byte sequences, enabling end-to-end tasks without intermediary subword or token representations. This paradigm is particularly pertinent for tasks where byte patterns encode critical information, such as software fuzz testing and language modeling over arbitrary digital inputs, and stands in contrast to standard fine-tuning over tokenized (e.g., BPE, unigram) text. Two principal domains—LLM-guided software fuzzing and LLM distillation—showcase the state of the art in byte-level supervised fine-tuning (Yang et al., 2024, Bao et al., 1 Feb 2026).

1. Formulation and Objectives

Byte-level supervised fine-tuning reframes the modeling objective from operating over token sequences to handling raw byte streams. The approach assigns the neural network a mapping from arbitrary-length byte sequences $x = (x_1, \dots, x_m)$ to task-specific targets, such as mutation instructions (Yang et al., 2024) or next-byte prediction (Bao et al., 1 Feb 2026). Supervision comes from ground-truth traces generated by upstream processes (e.g., fuzzer logs, byte-segmented text datasets), and optimization relies on standard cross-entropy between model outputs and these byte-level references.

In fuzz testing, this takes the form of conditional sequence-to-sequence learning: $$P(y|x) = P(p_1,s_1,\dots,p_k,s_k | x) = \prod_{j=1}^k P(p_j,s_j | x, p_{<j},s_{<j};\theta),$$ where $y=(p_1,s_1,\dots,p_k,s_k)$ encodes byte positions and mutation strategies (Yang et al., 2024). For general byte-LMs, the objective is

$$\mathcal{L}_{\rm SFT} = -\frac{1}{BL} \sum_{b=1}^B \sum_{i=1}^L \log \left[ \mathrm{Softmax}(\hat{y}_i^{(b)}) \right]_{x^{(b)}_i},$$

with $\hat{y}_i^{(b)}$ as logits predicting byte %%%%3%%%% from context $x_{<i}$ (Bao et al., 1 Feb 2026).

2. Data Construction and Preprocessing

Supervised fine-tuning at the byte level depends critically on dataset curation:

FuzzCoder / Fuzz-Instruct: Mutation logs are mined from a baseline fuzzer (AFL) performing coverage- or crash-inducing input mutations across diverse formats (ELF, XML, MP3, etc.). For each seed $x_i$, valid mutation traces $y_i$ are extracted as successful position-strategy pairs, yielding datasets with ≈30,000 examples and an 90/10 train/val split (Yang et al., 2024).

Distilled Byte LMs: Raw web corpora (FineWeb, 95 GB) are segmented into fixed-length (8192 bytes) windows, with the full 256-value byte vocabulary and no subword encoding. The entire document is processed without overlap, ensuring statistical diversity (Bao et al., 1 Feb 2026).

This reliance on true raw-byte representations, rather than textual tokens or characters, directly supports applications demanding arbitrary binary I/O handling and obviates the need for complex tokenization routines.

3. Model Architectures and Representation

Byte-level supervised fine-tuning leverages both off-the-shelf and specially adapted transformer-based architectures:

• Code LLMs in Fuzzing: Standard decoder-only Transformers (StarCoder-2, CodeLlama-7B/15B, DeepSeek-Coder-7B, CodeQwen-7B), with 32–48 masked attention layers, hidden size 4,096–8,192, rotary or ALiBi relative positional encoding. Byte-level BPE (BBPE) vocabularies (~50,000 tokens) are used, mapping each byte to one or two tokens (Yang et al., 2024).
• Distilled Byte LMs: Begins with a token-based backbone, then replaces the LM head with a “Dechunk” module (upsampling chunk embeddings to byte positions) and a byte-level decoder (predicting logits over all 256 bytes). After an initial head adaptation phase, the model becomes fully end-to-end in the byte domain (Bao et al., 1 Feb 2026).

Tokenization strategies differ by task: FuzzCoder employs BBPE for compatibility with code models, while byte-LMs eschew all subword processing, working on the native byte level.

4. Fine-Tuning Protocols and Hyperparameters

Distinctive training protocols characterize modern byte-level SFT:

FuzzCoder (Yang et al., 2024):

• Optimizer: AdamW ($\beta_1=0.9, \beta_2=0.95$), weight decay=0.1
• Learning rate: $5\times10^{-5}$, cosine decay, 3% warm-up
• Batch size: 1,024 (max length 4,096 tokens)
• Epochs: 3
• Adapter strategy: “mixture-of-adapters” per transformer block (~1% parameter overhead) to mitigate catastrophic forgetting
• Hardware: A100 40GB

Distilled Byte LMs (Bao et al., 1 Feb 2026):

• Step 1 (Head adaptation): Backbone frozen, train Dechunk + Decoder, LR $1\times10^{-3}$, warmup 10%
• Step 2 (Full fine-tuning): All weights unfrozen, LR $4\times10^{-5}$ for encoder/Dechunk/Decoder, $2\times10^{-5}$ for transformer core, warmup 1%
• AdamW optimizer, batch size 256 (8192 bytes), weight decay 0.1, max-grad-norm=1.0
• Scheduling: 95B bytes total (10% Step 1, 90% Step 2)

Pseudocode implementations for these schedules are available in the corresponding papers (Bao et al., 1 Feb 2026).

5. Quantitative Results and Empirical Impact

Byte-level supervised fine-tuning delivers empirical gains on core evaluation tasks as shown in the following tables.

FuzzCoder (Fuzzing Efficacy: EPM and NC)

Method	Avg. EPM (‰)	Avg. NC (1h)
AFL (original)	0.33	35
AFL + LSTM	0.54	19
AFL + Transformer	1.28	23
FuzzCoder (CodeQwen) 7B	1.69	70
FuzzCoder (DeepSeek-Coder) 7B	1.78	75
FuzzCoder (StarCoder-2) 15B	1.56	57

EPM: Effective Proportion of Mutations; NC: Number of Crashes (Yang et al., 2024)

These models more than double the effective mutation and crash discovery rates relative to heuristic baselines.

Distilled Byte LMs (Downstream Task Retention)

Stage	Data (B)	Avg Accuracy Drop	ARC-Challenge	PIQA	MMLU	HellaSwag
Stage 1	30 B	1.15	–	–	–	–
Stage 2 (SFT)	95 B	2.80	41.9	73.7	51.8	71.2

(Bao et al., 1 Feb 2026)

Despite the transition to pure byte-level generation, the accuracy drop remains modest—2.8 percentage points on average for a Llama-3.2 3B student, preserving over 92% of teacher competence.

6. Ablations, Insights, and Limitations

Key findings from ablation studies and protocol variations include:

• Traditional LSTM or Transformer models trained from scratch yield only marginal gains (EPM ≈0.5‰); pre-trained code LLMs fine-tuned at the byte level are substantially more effective (Yang et al., 2024).
• Larger model size (15B) does not guarantee superior performance; quality of pre-training corpus and coverage of byte-level instructions are critical.
• Adapter-only fine-tuning with frozen base parameters achieves >90% of the full model's gain in FuzzCoder, suggesting adaptation suffices for encoding most specialized logic.
• Byte-level SFT in LLMs preserves nearly all downstream capacity despite not relying on token-level context (Bao et al., 1 Feb 2026).

A plausible implication is that pre-trained model generalization, combined with judicious adapter or head specialization, underpins the success of byte-level SFT for non-textual and edge-case tasks.

7. Outlook and Future Directions

Potential research trajectories include:

• Reinforcement learning (RL) on coverage or downstream feedback, enabling LLMs to refine byte mutation policies via reward-maximizing online updates (Yang et al., 2024).
• Exploration of fully token-free (“raw byte”) LLM architectures to further reduce subword artifact impact.
• Application of chain-of-thought prompting for byte-level decompositions and complex structured mutations.
• Integration with symbolic execution frameworks to guide semantic mutations and bridge low-level byte alterations with high-level program semantics.
• Efficient curriculum and optimization methods for scaling byte-level SFT to trillion-byte or multi-lingual, multi-modal tasks (Bao et al., 1 Feb 2026).

This suggests that byte-level supervised fine-tuning is poised to play a central role in developing generalist models for domains with rich, arbitrary byte-sequence structure.