How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
Abstract: Synthetic data is a standard component in training LLMs, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop \textbf{\textsc{FinePhrase}}, a 486-billion-token open dataset of rephrased web text. We show that \textsc{FinePhrase} outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.
Summary
- The paper shows that structured pedagogical prompts—like math and FAQ formats—significantly improve the quality of synthetic pretraining data.
- It demonstrates that modestly scaled generator models can outperform larger ones, achieving up to 30-fold compute reductions while maintaining high performance.
- The study highlights the importance of mixing rephrased web data with high-quality original tokens to prevent model collapse and boost benchmark scores.
Systematic Investigation of Synthetic Pretraining Data: Prompt Design, Generator Scale, and Source Composition
Introduction
This work presents a comprehensive and controlled analysis of synthetic data synthesis pipelines for LLM pretraining, focusing on three key axes: prompt (output format) design, generator model characteristics, and data source selection. The central problem arises from the stagnation in the availability of high-quality, crawlable web text for pretraining, where recent approaches shift from massive data collection to compositional data transformation, primarily via rephrasing mechanisms implemented by LLMs. Large-scale empirical evaluations—over one trillion rephrased tokens—are conducted to uncover the optimal strategies for transforming web text into high-utility synthetic training samples. This work delivers actionable insights into the dominant factors of synthetic data efficacy and culminates with the construction and public release of FinePhrase, a 486-billion-token dataset, achieving strong benchmark results at a fraction of the resource cost of prior approaches (2604.13977).
Investigative Framework
The proposed experimental framework systematically controls three orthogonal variables in the synthetic data pipeline:
- Prompt Strategy: Both previously reported and newly proposed structured pedagogical prompts are evaluated. The new prompts instantiate outputs into math problems (with explicit solutions), FAQs, structured tables, and tutorials, driven by the empirical observation that output structure and educational content are strong signals for LLM acquisition.
- Generator Model Family and Scale: Synthetic data generation performance is analyzed across six model architectures, each scaled from sub-billion to tens of billions of parameters. Generator families assessed include SmolLM2, Gemma 3, Llama 3.2, Qwen 3, Granite 3.1, and Falcon 3.
- Source and Mix-in Data: Both source data for rephrasing and mix-in data for final training blends are factored, leveraging FineWeb subsets (partitioned by educational quality), DCLM, Cosmopedia, and other synthetic corpora. The pipeline enables disentangling source quality effects from those of the mix-in distribution.
Pretraining is performed with a 1.2B-parameter Qwen 2 model, and evaluation covers 12 benchmarks spanning factual, reading comprehension, reasoning, NLU, math, and tabular understanding. The default setting trains models on a 50/50 mix of synthetic and original tokens.
Empirical Findings
Impact of Prompt Format
Prompt format exerts the dominant influence on downstream task performance. Among established methods, prompts that impose structure—such as Diverse QA Pairs (Nemotron-CC)—exceed the curation or distillation of facts, but remain suboptimal compared to the newly introduced structured pedagogical forms. Math and table rephrasings induce statistically and practically significant improvements over both original DCLM web data and prior rephrasing methods.
Figure 1: Fine-grained evaluation of pedagogical formats against the DCLM baseline (macro-average scores, best marked with star); structure-driven rephrasings (math/table/faq/tutorial) consistently outperform baselines.
Structured math format achieves the highest aggregate benchmark scores, with table and FAQ versions closely following. Gains are reflected primarily in general knowledge and comprehension, with math outperforming DCLM by +1.54 macro-average and table by +1.06.
Generator Model Analysis
There is a strong ceiling effect for generator scale: models above 1B parameters do not yield better synthetic data, both in absolute benchmark scores and cost-adjusted returns. In many cases, the 1B-class generator outperforms considerably larger variants, inverts previously held assumptions about the necessity of extreme scale for effective rephrasing, and enables up to 30-fold compute reductions.
The model family further modulates output diversity and quality. SmolLM2 1.7B achieves the highest overall efficacy—especially in data diversity and reading comprehension benchmarks—pointing to the importance of instruction-tuning regime specificities rather than mere model size.
Source and Mix-in Data Composition
Mixing original web data with synthetic samples is essential: models trained exclusively on synthetic data underperform, particularly in linguistic diversity and NLU-oriented tasks, and are susceptible to model collapse. The optimal strategy is to rephrase both HQ and LQ web data using structured prompts and mix with a robust, high-quality corpus (DCLM or FineWeb-HQ), which can recuperate utility even from noisy seeds.
Crucially, data up-cycling via structured rephrasing yields competitive models when the mix-in is appropriately curated, relaxing quality constraints on source data and expanding the effective pool for large-scale pretraining.
Output Diversity, Template Collapse, and Pedagogical Trade-offs
Template collapse—where generator outputs are mechanically uniform—leads to lower downstream scores, even when instruction adherence is perfect. Output diversity, rather than rigid formatting, is required to avoid diminishing learning signals and model overfitting. SmolLM2's outputs are markedly more varied in length and form than those of similarly sized Qwen 3, correspondingly yielding higher benchmark performance.
The introduction of structured outputs incurs a trade-off: while general knowledge and analytical tasks improve, there remains a persistent deficit in linguistic and commonsense understanding (PIQA, HellaSwag) relative to raw web data. Synthetic signals and original web corpora are therefore synergistic in optimal pretraining pipelines.
Cost-Benefit Analysis and the FinePhrase Dataset
The FinePhrase dataset is constructed leveraging these findings: all samples are generated with SmolLM2 1.7B using the math, faq, table, and tutorial prompts, with completion tokens reaching 486B. The full pipeline achieves 9.2k tokens/s/GPU on H100 hardware, enabling a 30-fold generation cost reduction over prior strategies (e.g., REWIRE, Nemotron-CC) while improving absolute performance.
Figure 2: FinePhrase variants (structured pedagogical prompts using SmolLM2 1.7B, mixed with FineWeb-HQ) set new state-of-the-art performance on all macro-averaged benchmarks.
FinePhrase models outperform all tested baselines, highlighting the practical utility of the controlled ablation: performance improvements (up to +3.41 macro-average points) are achieved without resorting to extreme generator model scales, and all resources (data, code, ablation scripts) are shared for reproducibility.
Implications and Future Directions
The study conclusively demonstrates that prompt and data design dominate synthetic data generation effectiveness, with diminishing returns for larger generator models. Output diversity is not only desirable but required; rigid or consistently templated outputs harm model generalization. The potential to up-cycle lower-quality web data into valuable pretraining signals via structured transformation relaxes the emphasis on perfect curation, expanding available data reservoirs and reducing cost.
Practically, these results recalibrate resource allocation away from monolithic model scale increases and toward prompt engineering and more aggressive data recombination strategies. Theoretically, key open questions include the limits of prompt-driven up-cycling for extremely noisy data, optimal mixing ratios for high/low-resource languages, and further quantification of the interplay between structure and diversity in synthetic text. The template collapse phenomenon and the interaction with diverse instruction-tuned generators merit further algorithmic intervention.
Conclusion
This work represents the most systematic to-date analysis of synthetic pretraining data pipelines, isolating and quantifying the influences of prompt architecture, generator scale, and data mixture strategy. Structured pedagogical prompts, paired with modestly scaled generator models and careful mix-in with curated web tokens, produce both state-of-the-art benchmark results and orders-of-magnitude resource savings. FinePhrase operationalizes these principles, substantially raising the bar for open, reproducible, and efficient data synthesis in LLM development.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Plain‑Language Summary of “How Can We Synthesize High‑Quality Pretraining Data?”
What is this paper about?
This paper asks a simple question: if we want to make better training data for AI LLMs by rewriting (rephrasing) web pages, what’s the best way to do it? The authors run a large set of tests to find out which rewriting styles, which AI “writers,” and which types of source texts make the best “study material” for training new AI models.
What questions were the researchers trying to answer?
The team focused on three easy‑to‑grasp questions:
- Which way of rewriting text works best? (For example: turning articles into FAQs, tutorials, tables, or math problems.)
- How big and what kind of AI should we use to do the rewriting? (Small vs. big models; different model families.)
- Which data should we start from and mix in? (Does the quality of the original web text matter, and should we combine rewritten text with normal web text?)
How did they test this? (Methods in simple terms)
Think of this like training a student using different types of study guides and seeing which ones help them score higher on tests.
- They took lots of web text and gave “prompts” (instructions) to different AI “writers” to rewrite that text in various formats—like:
- FAQs (question-and-answer lists)
- Step‑by‑step tutorials
- Tables that neatly organize facts
- Math word problems with solutions
- They also tried well‑known earlier strategies (like summarizing, continuing a text, or making knowledge lists).
- They used different sizes of AI “writers,” from small to very large, to see if bigger writers produce better rewritten text.
- They then trained a new “student” AI from scratch on:
- The rewritten text alone, or
- A mix of rewritten text and normal web text.
- Finally, they tested the “student” AI on many tasks (like reading comprehension, general knowledge, and basic reasoning) to see which training recipe worked best.
Key idea: “Prompts” are simply the instructions given to the AI writer. “Tokens” are tiny pieces of text (like word chunks) used to measure data size. “Parameters” are a rough measure of an AI model’s size and capacity.
What did they find, and why is it important?
Here are the main takeaways, explained plainly:
- The format of the rewrite matters most. Turning web text into clear, structured study materials—especially tables, math problems (with step‑by‑step solutions), FAQs, and tutorials—consistently helps the “student” AI learn better than using plain web text or older rewriting styles. In many tests, these formats beat carefully curated web datasets and previous synthetic data methods.
- Bigger isn’t always better. Using a very large AI “writer” (more than about 1 billion parameters) didn’t improve the final “student” AI’s performance. In fact, smaller models often did just as well or better, which can save a lot of time and money.
- Mixing rewritten text with normal web text is crucial. Training only on rewritten (synthetic) text hurts the model’s understanding of everyday language and common sense. Mixing rewritten text with original web text gives the best results—rewrites provide clarity and structure, while real web text keeps language variety and “commonsense” knowledge.
- The dataset you mix in matters more than the source you rewrite. A strong mix‑in set of real web text boosts performance significantly, even if the original source you’re rewriting is lower quality. In other words, you can “up‑cycle” noisy web pages by rewriting them—so long as you also include a solid chunk of high‑quality real text in the mix.
- Variety beats rigid formatting. Some AI writers were too consistent, producing almost identical templates again and again (“template collapse”). That actually made the final “student” AI worse. More varied outputs—even if they weren’t perfectly formatted—led to better learning.
What did they build with these insights?
Using the best‑performing setup (small, efficient AI writers plus structured formats like tables, FAQs, tutorials, and math problems), the team created a new open dataset called FinePhrase:
- It contains 486 billion tokens (a very large amount of rewritten text).
- It outperforms existing synthetic data baselines on many tests.
- It was generated at up to 30 times lower compute cost than some previous methods (meaning it’s faster and cheaper to produce).
- They’re sharing the data, prompts, and code so others can use and improve them.
Why does this matter?
- Better training data means better AI models: clearer thinking, stronger reading comprehension, and improved factual knowledge.
- Lower costs mean more researchers (including smaller teams) can create strong training sets without needing huge budgets.
- The work offers a simple recipe: focus on structured, educational formats; use a small but capable AI writer; and always mix rewritten data with real web text to keep language skills diverse and natural.
In short, this paper shows that smart design choices—especially how you rewrite the text—matter more than brute force. With the right prompts and a good mix of data, even small, efficient models can help create high‑quality training data for the next generation of AI.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of concrete gaps and open questions the paper leaves unresolved; each item is framed to guide actionable follow-up work.
- Scaling of findings: Do the observed trends (prompt dominance, diminishing returns beyond ~1B generator params) hold when pretraining substantially larger target models (e.g., 7B–70B) and/or with much longer budgets (e.g., 100B–1T+ tokens)?
- Mixing ratio optimization: What are the optimal synthetic-to-web mixing ratios across training stages, model sizes, and domains? Is a curriculum (dynamic schedule over steps) superior to a fixed 50/50 mix?
- Generalization beyond English: How do results transfer to multilingual and low-resource languages (both as source and mix-in), and to cross-lingual rephrasing pipelines?
- Domain coverage: Do the conclusions extend to specialized domains (code, biomedical, legal, scientific writing, math beyond GSM8K) and domain-specific structured formats?
- Evaluation regime breadth: Do gains persist under zero-shot, chain-of-thought, and chat-style instruction evaluations, long-form generation, and human preference judgments?
- Long-context effects: How do longer context windows (e.g., 32k–200k) and chunking strategies affect the efficacy of structured prompts (especially tables/tutorials) and downstream long-context tasks?
- Target-architecture dependence: Do results replicate when the target model architecture differs (e.g., Llama, Mistral, Mamba, MoE) rather than only Qwen2-like at 1.2B?
- Statistical robustness: What is the variance across random seeds and data orderings? Which effects remain statistically significant with confidence intervals and multiple runs?
- Decoding and sampling ablations: How do temperature, top-p/top-k, repetition penalties, and speculative decoding configurations trade off diversity, correctness, and cost for synthetic data?
- Quantifying and mitigating template collapse: Can we define robust diversity metrics (n-gram novelty, structural entropy, prompt-shape variance) and intervention strategies (prompt randomization, multi-template ensembles, penalizing high-overlap outputs) to avoid collapse?
- Generator pretraining confounds: To what extent do generator instruction-tuning contents (e.g., rewrite-heavy data in SmolLM2) drive observed advantages? Controlled ablations with matched instruction corpora are needed.
- Multi-generator ensembles: Does mixing outputs from diverse generator families systematically improve diversity and downstream performance vs. any single generator?
- Prompt search and adaptation: Can automated prompt/program synthesis (e.g., evolutionary or Bayesian search) or document-adaptive prompting outperform the four hand-crafted pedagogical formats?
- Intra-format mixture design: What are the optimal proportions and scheduling among math, faq, table, and tutorial within FinePhrase-like mixtures?
- Source breadth and up-cycling limits: How far does “up-cycling” hold for extremely noisy sources (raw Common Crawl, forums, OCR’d PDFs), non-English sources, or domain-specific corpora (arXiv, PubMed, code repos)?
- Mix-in corpus design: Beyond FWHQ/DCLM, which mix-in types (Wikipedia-only, textbooks, news, academic, code) best restore NLU while preserving structured-data gains? Are domain-conditioned mix-ins beneficial?
- Dynamic mixing curricula: Do stage-wise curricula (early NLU-heavy mix-in, later structured-heavy synthetic) resolve the NLU vs. specialized-knowledge trade-off more effectively?
- Tokenizer sensitivity: How do different tokenizers (e.g., math- or code-aware) affect table/math format fidelity, compression efficiency, and downstream performance?
- Deduplication rigor: What is the impact of more aggressive deduplication (within synthetic, between synthetic and mix-in, and against web sources) on generalization and the observed gains?
- Contamination and decontamination: Are evaluation gains inflated by overlap with training data (especially Wikipedia-derived tasks)? A thorough decontamination audit is needed.
- Factuality and solution correctness: What fraction of synthetic math/FAQ/table/tutorial outputs contain factual or reasoning errors, and how do filtering/verification pipelines (e.g., solver checks) change outcomes?
- Safety, bias, and toxicity: How do different rephrasing prompts and generators shape bias/toxicity distributions, safety risks, and PII leakage in both synthetic data and trained models?
- Privacy and licensing: Do rephrasings inherit licensing/PII risks from sources, and what policy/technical mitigations (PII redaction, license tracking, watermarking) are effective at scale?
- Robustness and OOD behavior: How do models trained with structured synthetic data handle adversarial perturbations, distribution shift, noisy inputs, and calibration under uncertainty?
- Impact on downstream alignment: Does pretraining with structured synthetic data help or hinder subsequent SFT/RLHF, instruction following, and preference optimization?
- Broader benchmark coverage: Are gains consistent on larger and more diverse suites (e.g., full MMLU, BIG-bench Hard, TruthfulQA, DROP variants, code benchmarks)?
- Cost and energy portability: How do claimed cost advantages transfer across hardware (A100, consumer GPUs), software stacks, and energy/CO2 accounting; what are the Pareto frontiers under standardized conditions?
- Sequence-length constraints: The 4k-token training limit may truncate structured samples; what are the effects of alternative packing/chunking strategies on learning signals?
- Data quality scoring for synthetic outputs: Do educational-quality or factuality scores on synthetic data predict downstream gains, enabling better on-the-fly filtering and prioritization?
- Interaction effects between prompt type and mix-in: Mix-in studies used primarily the tutorial format; do optimal mix-ins change across math/faq/table, and are there prompt–mix-in synergies or antagonisms?
- Preservation of knowledge semantics: To what extent does rephrasing preserve fine-grained factual content and discourse structure vs. introducing distortions that affect knowledge probing later?
- Extending structure types: Are other pedagogical or structural schemas (flashcards, cloze deletions, knowledge graphs/JSON schemas, Socratic dialogues, executable tables) superior for specific competencies?
Practical Applications
Below is a distilled catalog of practical applications that flow directly from the paper’s findings on synthetic pretraining data design (prompting, generator choice, source/mix-in composition) and the FinePhrase dataset/pipeline. Each item notes what to build/do, why it works (the paper’s evidence), and the main dependencies/assumptions.
Immediate Applications
AI/Software Infrastructure and Product Teams
- Build a cost-efficient synthetic pretraining pipeline for domain LLMs
- What to build: A data-generation workflow using structured pedagogical prompts (math, table, FAQ, tutorial) with ~1B parameter instruction-tuned generators (e.g., SmolLM2 1.7B, Gemma 3 1B), mixed 50/50 with high-quality web tokens (e.g., FineWeb-HQ or DCLM).
- Why now: The paper shows structured formats consistently outperform web and prior synthetic baselines; small generators match/beat larger ones for synthesis; mixing synthetic with real text is necessary to maintain NLU and avoid collapse.
- Tools/workflow: DataTrove for orchestration, vLLM for distributed inference (+ speculative decoding), Nanotron for pretraining, LightEval for evaluation; use FinePhrase prompts or the open FinePhrase dataset (https://hf.co/datasets/HuggingFaceFW/finephrase) to bootstrap.
- Dependencies/assumptions: Access to licensed/allowable source corpora; GPU access (H100 or similar for reported throughput); English-first performance (paper scope); maintain diversity checks to avoid template collapse.
- Introduce “template collapse” monitors in data engineering pipelines
- What to build: A quality gate that flags over-regular patterns (identical openers, narrow length distributions, repeated scaffold texts) in generated corpora.
- Why now: The paper’s qualitative analysis shows diversity beats strict format obedience; over-consistent outputs hurt downstream performance.
- Tools/workflow: Simple n-gram and length-distribution checks; structure detectors to measure entropy across sections; integrate into CI for data pipelines.
- Dependencies/assumptions: Requires capturing generator logs and samples; thresholds tuned to domain norms.
- Mix-in dataset selection assistant
- What to build: A utility that chooses and validates the “real text” mix-in (e.g., DCLM/FineWeb-HQ) given your source corpus and rephrasing prompt choice.
- Why now: Mix-in choice is more critical than source data; high-quality mix-ins (DCLM/FWHQ) consistently deliver best outcomes.
- Tools/workflow: Scoring via a FineWeb-Edu-like classifier for mix candidates; report coverage, domain match, and quality scores.
- Dependencies/assumptions: Availability of high-quality mix-in corpora; clear licensing.
- Generate supervised QA and comprehension corpora at scale from proprietary documentation
- What to build: Automated creation of “Diverse QA Pairs” and FAQ/tutorial datasets from product manuals, wikis, or support forums for SFT and retrieval tuning.
- Why now: “Diverse QA Pairs” ranks top among legacy prompts; structured formats boost reading comprehension and factual tasks.
- Tools/workflow: Prompt library with Nemotron-CC QA + FinePhrase FAQ/tutorial prompts; apply to internal knowledge bases for enterprise chatbots.
- Dependencies/assumptions: Access to internal docs and data governance approval; SME review for sensitive domains.
Sector: Enterprise Knowledge Management & Customer Support
- Up-cycle noisy user-generated content into high-utility training data
- What to build: Rephrase forum posts and helpdesk transcripts into FAQ/tutorial/table formats; mix with high-quality web data to train “support LLMs.”
- Why now: The paper shows low-quality source can be “up-cycled” if paired with a strong mix-in; structured prompts deliver sizable gains.
- Tools/workflow: Automated ingestion → rephrase with ~1B model → diversity check → mix with FWHQ/DCLM → pretrain/continue-pretrain → RAG.
- Dependencies/assumptions: Consent and privacy controls; mix-in availability.
Sector: Education Technology
- Rapid generation of pedagogical corpora (problems, step-by-step tutorials, FAQs)
- What to build: Create large-scale math problems with solutions, structured tables, graded FAQs, and tutorials from open educational resources to pretrain/fine-tune teaching assistants.
- Why now: Math/table formats significantly outperform baselines, especially on reading comprehension and factual knowledge.
- Tools/workflow: FinePhrase prompts; curriculum design that balances formats; use mixed data to preserve NLU.
- Dependencies/assumptions: Instructor oversight to ensure alignment with standards; licensing for seed content.
Sector: Search/RAG and Data Platforms
- Structured-document converters for better indexing and downstream QA
- What to build: Services that transform unstructured web or enterprise content into tables and FAQs to improve retrieval and table-QA.
- Why now: Table format yields big gains on structured understanding; prompts explicitly create tables + Q&A from text.
- Tools/workflow: Rephrase to table + auto-generate Q/A; index both original and structured artifacts; evaluate with table QA.
- Dependencies/assumptions: Sufficient table coverage in domain; beware over-structuring rare content.
Academia and Open-Source
- Reproducible baselines for pretraining experimentation
- What to build: Course/research kits that ship FinePhrase prompts, small generators, and mix-in recipes to test pretraining hypotheses at modest compute.
- Why now: The paper releases data, prompts, code and shows small, reproducible configurations that rival large-model pipelines at a fraction of cost.
- Tools/workflow: Clone https://github.com/huggingface/finephrase and DataTrove examples; run on 1B-scale generators.
- Dependencies/assumptions: Compute modest but non-trivial; English-first; benchmark with LightEval.
Policy & Sustainability
- Best-practice guidance for energy-efficient synthetic data
- What to do: Draft internal or industry guidance recommending structured prompts + ~1B generators with mandated mix-in ratios to minimize energy use and avoid collapse.
- Why now: The paper shows up to 30× generation cost reduction vs. prior pipelines; purity (synthetic-only) underperforms.
- Tools/workflow: Standardized reporting (tokens/GPU-hour), mix-in ratio disclosure, and diversity audits.
- Dependencies/assumptions: Stakeholder buy-in; harmonization with data licensing and privacy policies.
Daily Life / Individual Practitioners
- Personal knowledge base “teaching-mode” transformation
- What to do: Convert notes, PDFs, and wiki pages into FAQs, tutorials, and tables for easier study and agent training (e.g., personal assistants).
- Why now: The structured formats are shown to increase comprehension-oriented performance.
- Tools/workflow: Run local ~1B instruction model to rephrase; index both structured and original notes; use for practice Q&A.
- Dependencies/assumptions: Local GPU/CPU capability; care with copyrighted material.
Long-Term Applications
Multilingual and Low-Resource Settings
- Extend structured rephrasing to non-English and mixed-script corpora
- Potential: Up-cycle low-quality content in low-resource languages by pairing with strong multilingual mix-ins; train domain LLMs where web text is scarce.
- Rationale: The paper’s “up-cycling” result plus mix-in dominance suggests feasibility beyond English.
- Dependencies/assumptions: High-quality multilingual mix-ins; culturally-appropriate prompts; evaluation benchmarks.
High-Stakes Domains (Healthcare, Finance, Legal)
- Domain-pretrained assistants built with structured rephrasing
- Potential: Rephrase clinical guidelines into tables/FAQs/tutorials; turn regulatory filings into structured corpora to pretrain safer, domain-specific LLMs.
- Rationale: Structured prompts improve factual/reading tasks; mixing keeps NLU; small generators reduce cost barriers.
- Dependencies/assumptions: Rigorous governance, licensing, and bias/safety audits; human validation of medical/legal correctness; regulator acceptance.
Robotics and Embodied AI
- Tutorial-style pretraining for procedural reasoning and planning
- Potential: Convert instructional texts into step-by-step tutorials for pretraining agents that follow long-horizon instructions.
- Rationale: Tutorial format strengthens sequential reasoning and instruction-following signals.
- Dependencies/assumptions: Multimodal grounding (text-to-action), real-world validation; expanded datasets.
Dynamic Curriculum and Auto-Mixing Policies
- Adaptive schedulers that tune format mix and mix-in ratios during training
- Potential: Systems that detect when to inject more NLU-rich web text vs. more structured pedagogical tokens to optimize convergence.
- Rationale: The paper identifies a trade-off between NLU and specialized knowledge; mixing restores balance.
- Dependencies/assumptions: Online measurement of task-specific metrics; auto-tuning policies; larger-scale experiments.
Data Marketplaces and Provenance Standards
- Commercial offerings of structured-rephrased corpora with provenance and diversity guarantees
- Potential: Sell vetted, licensed structured corpora and “diversity-certified” synthetic sets with template-collapse safeguards.
- Rationale: Clear efficiency and performance gains create market demand; reproducible pipelines enable verification.
- Dependencies/assumptions: Legal frameworks for licensing/rephrasing; provenance/hash verification standards.
On-Device/Edge Pretraining
- Private, small-footprint synthetic data generation for personalization
- Potential: Use ~1B generator locally to rephrase personal data into structured corpora, then continue-pretrain personal assistants.
- Rationale: Small generators are sufficient; structured prompts are strong; privacy preserved on device.
- Dependencies/assumptions: Efficient inference on edge hardware; robust template-collapse checks; battery/compute constraints.
Policy and Governance Instruments
- Synthetic data governance frameworks to mitigate model collapse and energy usage
- Potential: Require disclosure of synthetic ratios, mix-in sources, and diversity metrics; recommend small generators for generation tasks.
- Rationale: Findings confirm mix-in necessity and minimal gains from very large generators for rephrasing.
- Dependencies/assumptions: Cross-organization alignment; standard metrics (e.g., tokens/GPU-hour, diversity indices).
Cross-Modal and Structured-Knowledge Fusion
- Integrate structured rephrasing with knowledge graphs, tables, and code
- Potential: Hybrid corpora that combine LLM-friendly structured text with explicit graph/table representations to strengthen factual and procedural grounding.
- Rationale: Table/math formats already elevate structured reasoning; fusion could further boost capabilities.
- Dependencies/assumptions: Schema alignment; graph/table extraction at scale; revised tokenizers.
Notes on feasibility and dependencies common across applications:
- Mixing synthetic with high-quality natural text is a critical dependency to maintain NLU and avoid model collapse.
- The reported gains and costs are based on English corpora and specific hardware/software stacks (e.g., H100, vLLM with speculative decoding). Results may vary with other languages, models, and infrastructure.
- Legal, ethical, and licensing constraints must be honored when rephrasing source data, especially for enterprise and high-stakes domains.
- Diversity monitoring is essential to counteract template collapse; over-regular outputs can degrade downstream performance despite perfect instruction adherence.
Glossary
- ablation: An experimental method that removes or varies components to isolate their effects on outcomes. "we conduct a controlled ablation of the key components within the generation pipeline"
- AdamW: An optimizer that decouples weight decay from the gradient-based update for better regularization. "We use the AdamW optimizer~\citep{adamw} with a learning rate of , , , a weight decay of 0.1, and gradient clipping at 1.0."
- bfloat16: A 16‑bit floating-point format with the same exponent range as float32, enabling faster training with minimal loss of accuracy. "All runs leverage bfloat16 precision"
- cloze-format prompting: A prompting strategy where the model fills in masked or missing parts of text, often used for evaluation. "We employ 3-shot cloze-format prompting and report macro-averaged scores:"
- DataTrove: A data processing framework used to build large-scale ML pipelines. "We built our pipeline using DataTrove~\citep[v0.6.0]{datatrove}"
- entity-centric augmentation: Data generation focused on entities (e.g., people, places) to enrich pretraining corpora. "EntiGraph targets continued pretraining with entity-centric augmentation; their diversity scaling complements our finding that prompt-level diversity saturates at approximately 20B tokens."
- Flash Attention 2: An optimized attention algorithm that reduces memory usage and improves speed for transformer models. "All runs leverage bfloat16 precision, Flash Attention 2~\citep{flashattention2}, and rotary position embeddings~\citep{rope}."
- FineWeb-Edu classifier: A classifier that scores web samples by educational quality on a discrete scale. "The FineWeb-Edu classifier scores samples from 0 to 5 via Llama-3-70B-Instruct based on their educational quality."
- gradient clipping: A technique that limits gradient norms to stabilize training and prevent exploding gradients. "We use the AdamW optimizer~\citep{adamw} with a learning rate of , , , a weight decay of 0.1, and gradient clipping at 1.0."
- grouped-query attention: An attention variant where multiple query heads share key/value projections to improve efficiency. "Across all experiments, we train models with a 1.2B-parameter Qwen 2 architecture~\citep{qwen2} featuring ... through grouped-query attention~\citep{gqa}."
- instruction adherence: The degree to which generated outputs follow the given instructions or format. "the data of SmolLM2 yields superior downstream performance as observed in \S\ref{subsec:results_model}. We identify the cause as template collapse: a phenomenon where the extreme consistency of a generator produces repetitive and uniform training data that saturates the learning signal. For example, 115 outputs from Qwen 3 begin with identical introductory text. Conversely, the most frequent starting pattern in the data of SmolLM2 appears only three times. Furthermore, SmolLM2 shows a wider length variance (4 to 4,000 tokens) compared to the rigid formatting of Qwen 3 (100 to 2,600 tokens). These findings establish that for the objective of pretraining, the diversity of the output exerts a more substantial influence than the consistency of the format. A generator that is too well-behaved may inadvertently filter out the structural variety necessary for a robust signal, thereby exacerbating the narrow linguistic scope inherent to synthetic data (\S\ref{subsec:tradeoff})."
- instruction-tuned: Describes models fine-tuned to follow natural language instructions for tasks and formatting. "We construct the tested pretraining configurations from a rephrasing prompt (\S\ref{subsec:rephrasing}), an instruction-tuned generator model (\S\ref{subsec:generator})"
- LightEval: An evaluation toolkit for benchmarking LLMs. "and evaluation executed via LightEval~\citep[v0.9.1dev0]{lighteval}."
- macro-averaged scores: A scoring method that averages performance equally across tasks or classes, regardless of size. "We employ 3-shot cloze-format prompting and report macro-averaged scores:"
- mix-in dataset: The subset of original web data mixed with synthetic data during training. "the mix-in dataset is more critical than the source data;"
- model collapse: Degradation that occurs when models train primarily or exclusively on model-generated data, leading to loss of diversity and quality. "the model remains susceptible to the risk of model collapse~\citep{modelcollapse}"
- Nanotron: A training framework used for large-scale LLM pretraining. "training handled by Nanotron~\citep[v0.4]{ultrascale_playbook}"
- Pareto frontier: The set of configurations that are non-dominated in a trade-off between multiple objectives (e.g., cost vs. performance). "The Pareto frontier is dominated by small models employing the four structured prompts described in \S\ref{subsec:results_rephrase}."
- pedagogical format: A structured organization of content designed to enhance learning (e.g., tables, FAQs, tutorials, math problems). "We define a pedagogical format as a data structure that organizes information into discrete, logical, and queryable units."
- rotary position embeddings: A positional encoding technique that rotates query/key vectors to encode relative positions. "All runs leverage bfloat16 precision, Flash Attention 2~\citep{flashattention2}, and rotary position embeddings~\citep{rope}."
- scaling laws: Empirical relationships showing how performance scales with model size, data size, and compute. "This expansion followed scaling laws, which indicate that increased pretraining duration and data volume consistently improve downstream performance~\citep{kaplan2020scalinglawsneurallanguage,NEURIPS2022_c1e2faff}."
- speculative decoding: An inference acceleration method that uses a smaller draft model or strategy to propose tokens before verification. "By using 100 NVIDIA H100 GPUs and SmolLM2 1.7B with suffix-32 speculative decoding, the generation pipeline achieves a throughput of approximately 9,200 tokens per second per GPU."
- template collapse: A failure mode where generations become overly repetitive due to rigid adherence to templates, reducing training signal diversity. "template collapse: a phenomenon where the extreme consistency of a generator produces repetitive and uniform training data that saturates the learning signal."
- tokenizer: The algorithm that converts text into discrete tokens for model input and output. "For tokenization, we employ the Llama 3.2 tokenizer with a vocabulary size of 128,256."
- up-cycle: To convert low-quality inputs into higher-utility outputs through transformation (here, rephrasing). "the generation pipeline can successfully up-cycle noisy web text into high-utility training tokens"
- vLLM: A high-throughput inference engine optimized for LLM serving. "with distributed inference for synthetic data generation managed through vLLM~\citep[v0.8.4]{vllm}"
- weight decay: A regularization technique that penalizes large weights to improve generalization. "We use the AdamW optimizer~\citep{adamw} with a learning rate of , , , a weight decay of 0.1, and gradient clipping at 1.0."
Collections
Sign up for free to add this paper to one or more collections.