FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

Abstract: Due to limited supervised training data, LLMs are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .

• The paper introduces a pipeline that systematically generates billions of synthetic instruction-answer pairs from document-grounded templates for LLM pre-training.
• It employs semantic embedding and programmatic instantiation techniques to ensure diverse, high-quality data with over 80% source material accuracy.
• FineInstructions demonstrates improved benchmark performance across MixEval, MT-Bench-101, and AlpacaEval, enabling efficient scaling for smaller models.

FineInstructions: Instruction-Answer-Centric Scaling of Synthetic Data for LLM Pre-Training

Motivation and Context

Current LLM pre-training paradigms are dominated by next-token prediction over large corpora of unstructured text, followed by fine-tuning on orders-of-magnitude smaller supervised instruction-response datasets. The discordance between pre-training and downstream usage characteristics—especially the scarcity and narrowness of supervised instruction-tuning data—creates a potential inefficiency: vast compute budgets are directed toward absorbing knowledge through language modeling, not in formats that optimize LLM utility as instruction followers. This work introduces FineInstructions, a systematic pipeline and dataset that reconfigures pre-training at scale into a supervised-instructional framework, creating billions of synthetic, document-grounded instruction-answer pairs reflective of realistic user queries.

FineInstructions Pipeline and Dataset Construction

The core FineInstructions methodology proceeds as follows:

1. Instruction Template Generation: FineInstructions mines and genericizes ~18M user-written queries from diverse internet sources. Templates feature <fi> tags for variable insertion, yielding high task and domain diversity across generated instructions.
2. Document Matching via Semantic Embedding: Each instruction template is paired with compatible documents from large pre-training corpora using a two-stage, task-adapted semantic embedding model. A custom Gaussian pooling mechanism segments document embeddings to allow targeted, section-wise retrieval of candidate instructions for local document regions.
3. Programmatic Instantiation and Answer Extraction: Matching pairs are synthesized by instantiating templates with document-specific entities and extracting or slightly rephrasing passages to form answer candidates. Efficiency and answer veracity are enhanced by maximizing the inclusion of document-excerpted content—ensuring that ≥80% of response tokens reflect source material.
4. Filtered Quality Control: A judge model (Flow Judge, 3.8B params) assesses each instruction-answer instance according to a 5-point Likert scale, retaining only those with high relevance and directness (score ≥4).

Through recursive distillation (using Llama-3.3B as a base) and data-stratified filtering, the pipeline produces a large-scale, instruction-following dataset—over 1 billion high-quality instances—directly suitable for supervised pre-training.

Experimental Evaluation

FineInstructions is benchmarked against prominent baselines at scale—standard pre-training, Nemotron-CC [WRAP, Q&A], and Instruction Pre-Training (IPT)—under both equal token and compute budgets (23B and 300B tokens). Evaluation spans three widely recognized benchmarks:

• MixEval: Academic tasks and knowledge-intensive QA.
• MT-Bench-101: Realistic single-turn dialog, LLM-judged.
• AlpacaEval: Head-to-head LLM-judged, real-world query win-rates.

Key Numerical Results

• Superior Knowledge and Interaction Modeling: FineInstructions achieves MixEval accuracy improvements up to ~69% (IPT-scale) and ~39% (Nemotron-CC-scale) over standard pre-training, and outperforms all synthetic baselines across all LLM evaluation benchmarks.
• Efficient Scaling to Small Models: Models trained on FineInstructions data (e.g., 300M and 1.8B params) routinely match or surpass the benchmark performance of baselines trained at one model size higher, demonstrating substantial improvements in data and compute efficiency.
• Consistent Preference in Open-Ended Tasks: On AlpacaEval, FineInstructions-trained models win consistently in LLM-judged head-to-head settings, indicating robust generalization across both academic and free-form user-query domains.
• Quality Control Ablations: Addition of the automated judging and filtering stage yields further performance gains, especially on open-ended evaluation metrics.

Diversity and Structural Attributes

The instruction space exhibits notably high diversity:

• No individual template occupies more than 0.09% of generated instructions, with a long-tail distribution ensuring wide task coverage.
• Domain analysis finds representation across science, medicine, coding, reasoning, and open-ended/user-personalized queries.
• The complex matching mechanism allows instantiation of both generic and highly specific query types, enabling coverage of both broad and niche knowledge areas.

Implications and Future Directions

The work demonstrates that pre-training LLMs in an instruction-following regime at scale, using purely synthetic but document-grounded data, yields substantial improvements in downstream response quality, generalization to unseen instruction types, and data efficiency. This approach positions instruction-answer centric pre-training as a compelling alternative to standard language modeling, directly aligning models to target usage distributions.

On a practical level, the methodology allows for the generation of targeted specialist training corpora (e.g., domain-specific subsets), simplifies post-hoc adaptation to instruction tuning benchmarks, and reduces the necessity for expensive, manually curated instruction data. The document-grounded extraction mitigates risks of compounding hallucinations or systemic bias amplification inherent in monolithic LLM self-generation methods, although residual risks remain at scale.

Theoretically, these results suggest that curriculum and format alignment in pre-training objectives may yield sizable efficiency and generalization dividends for LLMs, particularly as smaller parameter models become increasingly attractive for deployment in resource-constrained environments.

Future work avenues include:

• Further optimizing instruction-document matching, template mixture weighting, and expansion to larger model scales.
• Exploring multi-turn dialog synthesis and task decomposition within the template/extraction paradigm.
• Developing new benchmarks that focus on the spectrum of long-tail, user-realistic knowledge and reasoning tasks, as current benchmarks are limited in this regard.

Conclusion

FineInstructions represents a scalable, systematic methodology for generating synthetic, high-diversity instruction-answer corpora grounded in naturalistic, document-based knowledge. When used for LLM pre-training, this approach markedly improves both knowledge absorption and task generalization under equivalent data and compute budgets, enabling the efficient training of smaller, high-performance instruction-following models. The paradigm shift from next-token prediction to instructional supervision at scale opens new trajectories for both the practical efficiency and theoretical underpinnings of LLM pre-training and alignment (2601.22146).