Papers
Topics
Authors
Recent
Search
2000 character limit reached

Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

Published 7 May 2026 in cs.AI and cs.CL | (2605.06638v1)

Abstract: Reinforcement learning (RL) has been applied to improve LLM reasoning, yet the systematic study of how training scales with task difficulty has been hampered by the lack of controlled, scalable environments. We introduce ScaleLogic, a synthetic logical reasoning framework that offers independent control over two axes of difficulty: the depth of the required proof planning (i.e., the horizon) and the expressiveness of the underlying logic. Our proposed framework supports a wide range of logics: from simple implication-only logic ("if-then") towards more expressive first-order reasoning with conjunction ("and"), disjunction ("or"), negation ("not"), and universal quantification ("for all"). Using this framework, we show that the RL training compute $T$ follows a power law with respect to reasoning depth $D$ ($T \propto Dγ$, $R{2} > 0.99$), and that the scaling exponent $γ$ increases monotonically with logical expressiveness, from $1.04$ to $2.60$. On downstream mathematics and general reasoning benchmarks, more expressive training settings yield both larger performance gains (up to $+10.66$ points) and more compute-efficient transfer compared to less expressive settings, demonstrating that what a model is trained on, not just how much it is trained, shapes downstream transfer. We further show that the power-law relationship holds across multiple RL methods, and curriculum-based training substantially improves scaling efficiency.

Summary

  • The paper demonstrates that RL post-training on synthetic logical tasks follows a power-law sample complexity, scaling with proof depth and logical expressiveness.
  • It reveals that enhanced logical operators, notably quantification, boost downstream transfer gains by up to 10.66 points on real-world benchmarks.
  • Curriculum training reduces scaling exponents, emphasizing that gradual difficulty progression is key to achieving robust long-horizon reasoning.

Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key

Introduction and Motivation

The paper "Can RL Teach Long-Horizon Reasoning to LLMs? Expressiveness Is Key" (2605.06638) systematically investigates the scaling behavior of reinforcement learning (RL) post-training for LLMs on long-horizon reasoning tasks, emphasizing the role of logical expressiveness in determining both the sample efficiency and the effectiveness of downstream reasoning transfer. The lack of controlled, scalable environments for fine-grained analysis of reasoning complexity has hindered the empirical study of RL scaling in this domain.

This work introduces ScaleLogic, a synthetic logical reasoning framework with precise control over two orthogonal axes of task difficulty: the horizon (proof depth) and the logical expressiveness (available logical operators and quantification). ScaleLogic enables data generation with exact verifiability, variable reasoning horizon, and tunable logical structure, creating a testbed for empirical scaling-law studies and for developing RL-based training protocols targeting generalizable reasoning skill in LLMs. Figure 1

Figure 1: Overview of the ScaleLogic task construction regime, ranging from simple implication-only instances to highly expressive problems requiring quantification, conjunction, disjunction, and negation.

Methodology: ScaleLogic Environment and Control Axes

ScaleLogic procedurally generates multi-choice logical reasoning instances. Each instance provides a set of axioms (expressed as facts and rules in natural language) and several candidate conclusions. Exactly one candidate is logically entailed; the rest are made unprovable by minimal corruptions to their proof trees. The key design allows two explicit task variables:

  • Reasoning Depth (DD): Regulates proof tree depth, governing the "horizon" of necessary multi-step inference.
  • Logical Expressiveness: Follows a strict hierarchy:

    1. Implication-only (basic if-then),
    2. +Conjunction (and in antecedents),
    3. +Negation (not in premise/conclusion),
    4. +Disjunction (or in consequents),
    5. +Quantification (universal quantifiers over entities).

This decomposition isolates the effect of fundamental logical constructs. By independently varying DD and expressiveness, the study rigorously disentangles the impact of structural complexity from surface-level instance difficulty.

Scaling Laws: Training Compute vs. Task Complexity

The central empirical finding is that, for RL post-training, the training effort TT (measured as RL steps to reach a fixed accuracy) obeys a power law in DD,

T∝Dγ,T \propto D^\gamma,

with R2>0.99R^2 > 0.99 across all five expressivity settings. Crucially, the scaling exponent γ\gamma increases monotonically with logical expressiveness, ranging from $1.04$ (Implication-only) up to $2.60$ (+Quantification). This demonstrates that richer logical operators, even at the same horizon, impose a strictly superlinear increase in RL sample complexity. Figure 2

Figure 2

Figure 2: Log-log scaling of training steps versus depth, with power law fits; higher logical expressiveness settings exhibit steeper exponents.

The paper contrasts this with exponential fits, which are systematically outperformed in data-fit metrics. This implies that, within the tested regime, long-horizon reasoning cost increases polynomially, but at a rate acutely sensitive to the logical structures the model must master.

Downstream Transfer: Beyond Synthetic Tasks

The study further evaluates whether RL on synthetic logical tasks transfers to real-world mathematical and reasoning benchmarks (AIME, AMC, MATH-500, Minerva, OlympiadBench, GPQA-Diamond, MMLU-Pro). The strongest downstream gains occur in the most expressive synthetic training environments:

  • +Quantification yields mean downstream accuracy improvements up to +10.66 points over the base model.

  • Simple logic settings lead only to marginal or early-peaking gains. Figure 3

Figure 3

Figure 3: Downstream benchmark performance as a function of RL post-training steps under different logic expressiveness conditions; curves for richer logic exhibit monotonic improvement, in contrast to plateauing for simple settings.

When controlling for both train compute and fixed proof depth, higher expressivity strictly yields stronger transfer, indicating that what the model is trained on (in terms of logical structure) is more predictive of real-world reasoning gains than the aggregate amount of compute.

Effect of the Training Distribution and RL Method

The training distribution over proof depths has a marked effect on learning efficiency:

  • Curriculum training (gradual progression to deeper tasks) yields strictly lower scaling exponents compared to uniform or difficult-only distributions, e.g., lowering γ\gamma from DD0 (uniform) to DD1 (+Quantification).
  • All tested RL algorithms (DAPO, GRPO, GSPO) follow the same power law scaling regime, differing only in fitted exponents, validating the robustness of the phenomenon across common RL optimization recipes. Figure 4

Figure 4

Figure 4: Curriculum learning improves scaling; training distributions that emphasize gradual depth increase produce lower power-law exponents.

Generalization and Limits of Long-Horizon Reasoning

By evaluating OOD generalization (train at depth DD2, test at depth DD3), the experiments show that:

  • Increasing training depth extends the model's effective reasoning horizon linearly, but generalization is bounded—accuracy rapidly degrades past approximately DD4 trained depth.
  • Scaling up candidate count DD5, while increasing computational cost (DD6), yields only modest downstream gains, implying that depth is the more potent axis for developing transferable reasoning skill. Figure 5

Figure 5

Figure 5: Validation accuracy as a function of absolute test depth for models trained to various maximum depths; OOD performance degrades predictably beyond the seen horizon.

Performance Comparison Against Frontier Models

Evaluations of high-profile LLMs (GPT-4o, DeepSeek V3.1, Qwen3.5) on the most expressive ScaleLogic tasks reveal a common phenomenon: all models exhibit rapid accuracy decay with increasing reasoning depth, highlighting the unique challenge posed by the synthetic environment and the necessity of targeted RL post-training for pushing long-horizon reasoning limits. Figure 6

Figure 6: Number of correct answers for state-of-the-art LLMs on ScaleLogic as a function of proof depth, showing rapid decline beyond DD7 in expressive settings.

Implications, Theoretical and Practical

Empirical Claims:

  • Power-law sample complexity: RL on reasoning tasks is best characterized by a power law in horizon, but the slope of this law is a function of logical expressiveness.
  • Expressiveness governs transfer: Downstream performance depends not just on training compute, but on the presence of rich logical operators and quantification in synthetic training data.
  • Curriculum improves scaling: Progressive data regimes improve sample efficiency, suggesting links to compositional generalization.

Theoretical Implications:

  • The clean dependence of DD8 on logical expressiveness calls for formal analysis: why do specific operators (e.g., quantification, disjunction) so strongly alter sample complexity? Connections to structural properties of deductive search spaces and the compositionality of RL in sequence models merit theoretical exploration.

Practical Directions:

  • For practitioners aiming to elicit advanced, generalizable reasoning from LLMs, increasing logical expressiveness in synthetic RL training pipelines is essential.
  • Power-law analysis provides a principled metric for analyzing and comparing RL post-training regimes beyond simple step- or token-based reporting.
  • The bounded horizon generalization emphasizes the need for both architectural and objective innovation to break through persistent limitations in chain-of-thought reasoning depth.

Conclusion

This work establishes a new experimental paradigm for measuring and improving long-horizon reasoning in LLMs via controlled RL post-training. The central finding—that logical expressiveness is the dominant factor controlling scaling efficiency and downstream transfer—has immediate methodological implications for the design of future reasoning-oriented RL pipelines. The results motivate both further scaling studies across model sizes and the development of training curricula and synthetic environments targeting the most challenging fragments of logic. The observed power-law regime and expressiveness dependence highlight the need for both empirical and theoretical advances to enable truly robust long-horizon reasoning in LLMs.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What is this paper about?

This paper asks a simple, big question: Can we train AI LLMs to think through long, multi-step problems using reinforcement learning (RL)? To study this clearly, the authors build a new “playground” of logic puzzles called ScaleLogic, where they can dial up how hard the puzzles are in two ways: by making the solution require more steps (longer “horizons”) and by using richer kinds of logic (like adding “and,” “or,” “not,” and “for all”). Then they measure how much training it takes for models to get good at these puzzles, and whether that training helps on real math and reasoning tests.

What questions did the researchers ask?

  • How does the training effort needed to reach good accuracy grow as problems need more reasoning steps?
  • Does making the logic more expressive (from just “if-then” up to “for all”) change how training scales?
  • Do skills learned on these synthetic logic puzzles transfer to real-world math and reasoning tests?
  • Do different RL methods show the same patterns?
  • Can smart training choices (like a curriculum from easy to hard) make learning more efficient?

How did they study it?

Think of ScaleLogic like a factory that makes logic puzzles with two knobs:

  1. Depth (horizon): how many steps are required to reach the answer—like the number of links in a chain.
  2. Expressiveness: how rich the logic is. The researchers add these features one by one:
    • Implication-only: just “if A then B.”
    • + Conjunction: “and.”
    • + Negation: “not.”
    • + Disjunction: “or.”
    • + Quantification: “for all” (rules that apply to anyone, not just a specific person).

Each puzzle gives you:

  • A set of facts and rules (the “axioms”).
  • Several candidate conclusions (multiple choice), but only one is actually provable from the facts.

To make sure the model can’t rely on cheap tricks, the authors carefully build each wrong option by “breaking” exactly one needed fact in its proof. That way, the model has to truly reason about which conclusion can be derived.

Reinforcement learning setup:

  • The model tries to answer the multiple-choice question.
  • If it picks the logically correct option in the required format, it gets a reward (1 point); otherwise, 0.
  • Over many puzzles, the model learns which kinds of reasoning steps lead to correct answers.
  • They also test a “curriculum”: starting with easier (shallower) puzzles and gradually increasing difficulty.

What did they discover?

Here are the key findings, in plain terms:

  • Training follows a power law as puzzles get deeper. That means if you measure the number of RL training steps needed to reach 90% accuracy (call this T) and the required depth (call this D), then T grows like a power of D (not explosively, but steadily). In words: as you double the number of reasoning steps, the training cost increases in a predictable way.
    • In the simplest logic (just “if-then”), doubling depth roughly doubles the needed training.
    • In the richest logic (with “and,” “or,” “not,” and “for all”), doubling depth can make training take about six times longer.
    • This “how quickly it grows” is captured by an exponent called γ (gamma). It increases as the logic gets more expressive: from about 1.04 (simple) up to about 2.60 (richest).
  • This pattern shows up across different RL methods. No matter which RL algorithm they used, the “power law” scaling appeared.
  • A curriculum helps a lot. Training from easy to hard reduces how fast the training cost grows with depth, making learning more efficient and stable.
  • The benefits transfer to real benchmarks. Models trained on the most expressive logic puzzles improved the average score on real math and reasoning tests by up to +10.66 percentage points over the base model. Simpler training settings gave smaller gains and often plateaued early; the richer logic kept improving with more training.
  • Training at depth D helps on harder tests, but only up to a point. If you train at a certain depth, you can handle tests up to about three times that depth before performance drops to chance. So training stretches the “reasoning horizon,” but it doesn’t make it unlimited.

Why this matters: The results show that what you train on really matters. Training on richer logic isn’t just harder—it teaches skills that carry over better to real problems and uses compute more effectively as you scale up.

Why does this matter?

  • A clean testbed for long reasoning: ScaleLogic gives researchers a controlled way to dial up difficulty and see how RL behaves, without messy data issues. That helps us understand how to build better reasoning training for AI.
  • Training content > just more training: It’s not only about doing more training steps. Choosing richer, more expressive problems leads to bigger, more lasting improvements on real tasks.
  • Practical guidance: If you want stronger reasoning in LLMs, train on:
    • Verifiable tasks (so rewards are reliable),
    • Rich logical structures (so models learn general skills),
    • A curriculum (so they learn efficiently).
  • Future directions: Try this with larger models, add even richer logic (like equality or more complex real-world relations), and develop theory to explain why the scaling exponents change with logical expressiveness. This could lead to principled recipes for teaching AI to reason over very long chains of thought.
View Paper Prompt  View All Prompts 

Knowledge Gaps

Below is a single, focused list of the paper’s unresolved knowledge gaps, limitations, and open questions to guide future research.

  • Model scale: Results are primarily on Qwen3-4B (with limited 8B checks). It is unknown whether the power-law scaling and the monotonic increase of the exponent γ\gamma with expressiveness persist for much larger models (e.g., ≥70B) and different base model families.
  • Compute metric: Training “compute” is proxied by RL steps (TT). The invariance of conclusions under more faithful compute measures (tokens, FLOPs, wall-clock) and under different sequence lengths, group sizes, or sampling temperatures remains untested.
  • Depth range and threshold sensitivity: Power-law fits are established over a finite depth range and at a 90% accuracy threshold. How γ\gamma varies with higher depths, alternative thresholds (e.g., 70%, 95%, 99%), and alternative stopping criteria is not characterized.
  • Branching factor (BB): Most analyses fix B=4B=4; joint scaling in both DD and BB (including interactions) is largely unexplored, leaving open whether γ\gamma depends systematically on BB.
  • Training data volume: Dataset size is typically fixed (∼100k instances). How scaling changes with data volume (data-scarce vs data-rich regimes) and repeated exposure (multi-epoch training) is not examined.
  • Curriculum design space: Only depth-based curricula are explored. Whether curricula based on expressiveness, mixed difficulty, operator composition, or adaptive/competence-based schedules further reduce γ\gamma is unknown.
  • Algorithmic breadth: Scaling is tested with GRPO, DAPO, and GSPO. The behavior under other RL families (e.g., actor-critic with learned value baselines, advantage estimation variants, offline RL, evolutionary methods, DPO/ReFT-style objectives) is open.
  • Reward shaping: Rewards are binary final-answer verifications. It is unclear how intermediate/shaped rewards (e.g., stepwise proof verification, partial credit) affect sample efficiency, stability, and γ\gamma.
  • Decoding/test-time scaling: The interaction between RL-learned reasoning and test-time compute (self-consistency, tree search, beam search) is not analyzed; whether RL and test-time scaling combine additively or redundantly is unknown.
  • Proof uniqueness constraint: Instances enforce unique derivations via fresh predicates. The impact of multiple valid proof paths, cycles, or richer dependency graphs on scaling and γ\gamma is untested.
  • Logical fragment restrictions: Conjunction is restricted to antecedents, disjunction to consequents; no existential quantifiers, equality, functions, higher-order logic, or proof by contradiction are included. How these operators alter scaling and transfer remains open.
  • Negation and inconsistency: Negation is included, but settings avoid inconsistent axioms and paraconsistent regimes. How contradictions, closed-world assumptions, or non-monotonic reasoning affect learning and γ\gamma is unknown.
  • Quantification scope: Only universal quantification with simple instantiation is used. The effects of variable unification challenges, existential quantifiers, and multi-variable relational constraints are unstudied.
  • Multi-entity relational structure: Proof trees largely avoid predicate reuse to ensure uniqueness. The impact of dense relational graphs with entity/predicate reuse and cross-entity constraints (more realistic KBs) is not evaluated.
  • Surface form realism: Natural language is template-based with fictional predicates. Robustness to paraphrase, lexical ambiguity, coreference, morphological/syntactic variation, and longer multi-sentence contexts is not assessed.
  • Heuristic exploitation: Although corruption is uniform, potential surface heuristics (positioning, lexical cues, distractor distribution) are not stress-tested with adversarial or counterbalanced designs.
  • Distractor rules: “Local ambiguity” distractors are introduced, but their quantity, placement, and types are not systematically ablated to quantify their contribution to difficulty and scaling.
  • Open-ended outputs: Training/evaluation are multiple-choice. Whether the scaling laws and transfer hold for open-ended proof generation (with formal proof checkers or free-form CoT scoring) remains unverified.
  • Downstream breadth: Transfer is shown mainly on math/STEM-heavy benchmarks. Effects on commonsense entailment, multi-hop QA, planning, code reasoning, and non-STEM domains (and multilingual settings) remain open.
  • Mechanistic understanding of transfer: The causal link between specific logical operators in training and downstream gains is not probed (e.g., operator-frequency ablations, diagnostic tasks to localize learned skills).
  • OOD horizon limits: OOD gains saturate near Dtest≈3DtrainD_{\text{test}}\approx 3D_{\text{train}} under +Quantification; how this frontier shifts with larger models, stronger curricula, different BB, or different operator mixes is unknown.
  • Stability and variance: Greater variance appears at high depths and in difficult-only training; a thorough characterization of convergence failures, instability modes, and hyperparameter sensitivity is missing.
  • Pretraining interactions: How base pretraining (e.g., math/code-heavy vs general text) modulates RL scaling, γ\gamma, and transfer has not been systematically compared across base models.
  • Context length effects: As DD increases, context length and memory demands rise; the dependence of scaling on context window size, retrieval, or memory-augmented architectures is unstudied.
  • Noisy/contradictory environments: Robustness to label noise, adversarial axioms, or partially incorrect/irrelevant rules (beyond current distractors) is not quantified.
  • Compute-efficiency claims: “More compute-efficient transfer” is argued qualitatively; normalized comparisons by tokens/FLOPs/wall-clock across settings and hardware to substantiate efficiency claims are limited.
  • Theoretical account: There is no formal theory explaining why γ\gamma increases with expressiveness or predicting the observed values; linking operator combinatorics, sample complexity, and RL dynamics remains an open theoretical problem.
  • Agentic/tool use settings: It is unknown whether ScaleLogic training benefits interactive, tool-augmented agents (retrieval, calculators, theorem provers) and how expressiveness interacts with tool-use policies.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are actionable use cases that can be deployed with current methods and infrastructure, derived from the paper’s findings on ScaleLogic, RL with verifiable rewards, power‑law training compute, and curriculum strategies.

  • Software/AI (model training): Expressiveness‑controlled synthetic RLVR training data to improve reasoning
    • What: Use ScaleLogic to generate verifiable, low‑cost training data with adjustable depth and logical operators (+conjunction/negation/disjunction/quantification) to post‑train small/medium LLMs via GRPO/DAPO/GSPO.
    • Tools/workflows: “Synthetic Verifiable Data Generator” API with knobs for depth D, candidates B, and operator set; plug into existing RLVR pipelines.
    • Assumptions/dependencies: RLVR infrastructure; simple binary reward format; observed transfer (+10.66 points) replicates for your base model/domain.
  • MLOps (training efficiency): Compute forecaster and curriculum scheduler for long‑horizon reasoning
    • What: Fit T∝DγT \propto D^{\gamma} to your training runs to forecast compute/time-to-target accuracy; deploy curriculum sampling (shown to reduce γ and variance) to improve scaling efficiency.
    • Tools/workflows: “Compute Forecaster” dashboards; “Reasoning Curriculum Scheduler” that increases D as validation improves.
    • Assumptions/dependencies: Accurate estimation of γ for your model/logic mix; online monitoring of Pass@1.
  • QA/Benchmarking (industry and academia): Acceptance tests for long‑horizon reasoning robustness
    • What: Add ScaleLogic‑style evaluations with increasing D and expressiveness to pre‑deployment QA for assistants, code agents, and planning agents.
    • Tools/workflows: “Reasoning Benchmark Suite” with tiered gates (e.g., must pass D≤k under +Quantification).
    • Assumptions/dependencies: Benchmarks reflect task horizon; OOD limit (performance drops when D_test/D_train ≈ 3) used to set safe thresholds.
  • Product assistants (customer support/IT): Finetuning and regression tests for decision‑tree/diagnostic flows
    • What: Train and test chatbots on implication/+conjunction instances to improve multi‑step follow‑through and reduce “shortcut” heuristics.
    • Tools/workflows: Domain templating of predicates; periodic regression tests at fixed D.
    • Assumptions/dependencies: Template mapping from domain rules to natural language; verifiers match output format.
  • Software engineering: Sanity checks for code agents’ multi‑step reasoning
    • What: Before shipping, validate and tune agents using increasingly expressive synthetic logic tasks to ensure stable chain‑of‑thought at longer horizons.
    • Tools/workflows: “Horizon Gate” in CI/CD; failure triage by depth and operator set.
    • Assumptions/dependencies: Mapping from logical failures to agent prompts/policies; RLVR or supervised corrective finetuning.
  • Finance/Compliance: Pretraining on logic with negation/conjunction for rule adherence and exception handling
    • What: Use synthetic tasks with negation (e.g., policy disallowances) to train models to honor constraints and handle mutually exclusive conditions.
    • Tools/workflows: Domain lexicons for policies; holdout tests at raised D for policy chains.
    • Assumptions/dependencies: Strong domain validation; synthetic‑to‑real generalization may vary—treat as pretraining signal, not sole assurance.
  • Robotics/Planning (simulation): Planner bootstrapping with verifiable long‑horizon chains
    • What: Train text‑based planners or language‑conditioned policies on long proof‑like chains to strengthen multi‑step precondition satisfaction.
    • Tools/workflows: Sim‑to‑real curriculum: start in synthetic logic, then blend with environment‑specific verifiable tasks.
    • Assumptions/dependencies: Reward design for downstream planning tasks; transfer from symbolic logic to environmental dynamics may be partial.
  • Education: Logic/LSAT‑style practice sets with controlled difficulty and automatic grading
    • What: Offer students instances at targeted depths and operator mixes, with exact verification and adaptive curricula.
    • Tools/workflows: “Adaptive Logic Tutor” using D‑progression and expressiveness ramps.
    • Assumptions/dependencies: High‑quality natural language templates; content alignment with curricula/exams.
  • Risk management (all sectors): Horizon guardrails for deployed assistants
    • What: Estimate the trained horizon and enforce task segmentation if a user request implies D beyond a safety margin (e.g., ~2–3× trained depth).
    • Tools/workflows: “Horizon Estimator” and “Task Splitter” that decomposes requests into shallower steps.
    • Assumptions/dependencies: Reliable heuristics to approximate task horizon; human oversight for high‑stakes tasks.
  • Research (academia/AI labs): Reproducible scaling‑law studies and algorithm comparisons
    • What: Use ScaleLogic to study how model size, RL method, and training distribution affect γ and transfer.
    • Tools/workflows: Open pipelines to reproduce power‑law fits; ablations over operators and curricula.
    • Assumptions/dependencies: Access to training infrastructure; sharing of seeds/configs for comparability.

Long‑Term Applications

These opportunities require further research, scaling, domain integration, or validation before deployment.

  • Healthcare: Clinical reasoning and guideline adherence with expressive logical training
    • What: Train CDS assistants to handle multi‑premise, exception‑laden guidelines (conjunction/negation/quantification).
    • Potential products: “Guideline Tutor” and “Order Set Checker” with proof‑structured rationales.
    • Assumptions/dependencies: Rigorous clinical validation; medical data integration; safety and regulatory approval; verifiable reward design for clinical tasks.
  • Legal/Policy: Long‑horizon legal reasoning assistants and compliance auditors
    • What: Combine expressive logical training with domain knowledge to trace statutes, precedents, and exceptions across steps.
    • Potential products: “Compliance Chain Analyzer”; “Precedent Path Explorer.”
    • Assumptions/dependencies: Accurate legal knowledge bases; human oversight; jurisdictional variation; liability management.
  • Autonomous agents: Robust multi‑step planning and tool use under horizon guarantees
    • What: Train agents on increasingly expressive synthetic tasks, then scaffold real‑world workflows to match trained horizons.
    • Potential products: “Horizon‑aware Agent Orchestrator” with dynamic task decomposition.
    • Assumptions/dependencies: Reliable horizon estimation in the wild; strong transfer from synthetic logic to embodied/action domains.
  • Standards and certification: Sector‑wide benchmarks for long‑horizon reasoning
    • What: Use depth/expressiveness‑controlled suites as part of pre‑deployment certification in safety‑critical sectors.
    • Potential products: “Reasoning Maturity Levels” (RMLs) keyed to D and operator sets.
    • Assumptions/dependencies: Multi‑stakeholder adoption; governance processes; mapping benchmarks to real risk profiles.
  • ScaleLogic‑as‑a‑Service: Enterprise platform for synthetic verifiable data and training analytics
    • What: Hosted generation with per‑tenant templates and “compute forecaster” features; auto‑curriculum recipes for RL.
    • Potential products: Dataset generator, analytics dashboard, curriculum manager.
    • Assumptions/dependencies: Platform reliability; IP/privacy for domain templates; integration with varied training stacks.
  • Formal methods integration: Bridging RL‑trained reasoning with proof assistants and verifiers
    • What: Train models to produce proof objects and check them with external engines; extend expressiveness to equality/higher‑order logic.
    • Potential products: “Proof‑aware CoT” and “Proof‑checked Agents.”
    • Assumptions/dependencies: Verifier interfaces; reward shaping for proof objects; scalability to richer logics.
  • Domain‑specific synthetic environments: Multi‑entity relational and temporal reasoning
    • What: Extend expressiveness to capture relations, quantification over sets, temporal/event logic for operations planning.
    • Potential products: “Relational Reasoning Gym” for supply chain, incident response, or cybersecurity.
    • Assumptions/dependencies: Task‑specific verifiers; careful control of expressiveness to manage γ and compute.
  • Compute‑aware training pipelines at scale: Organization‑wide budgeting and scheduling
    • What: Institutionalize power‑law compute planning and curriculum across teams to reach target accuracy at minimal cost.
    • Potential products: “Org‑level Compute Planner” that allocates GPUs by projected γ and D.
    • Assumptions/dependencies: Reliable historical fit of γ across models; shared telemetry.
  • Test‑time compute controllers: Dynamic CoT budgeting using learned exponents
    • What: Use estimated γ to set step budgets and branch factors for tree‑of‑thought methods, balancing latency and accuracy.
    • Potential products: “Reasoning Budget Controller” for production inference.
    • Assumptions/dependencies: Stable relationship between horizon and required test‑time compute; monitoring for drift.
  • Theory‑informed algorithm design: Methods that lower γ via curricula, representations, or architectures
    • What: Develop RL objectives, memory mechanisms, or symbolic‑neural hybrids that flatten scaling curves at higher expressiveness.
    • Potential products: New optimizers and modules targeting conjunction/disjunction/quantification handling.
    • Assumptions/dependencies: Cross‑model validation; broader tests beyond synthetic settings.

Cross‑cutting assumptions and dependencies

  • Transfer limits: OOD generalization is bounded; accuracy drops near D_test/D_train ≈ 3. Production systems should align task horizons with training or apply decomposition.
  • Reward design: Immediate binary rewards work for multiple‑choice; many real tasks need richer verifiers or proxy rewards.
  • Data‑to‑domain mapping: Template quality and predicate semantics must reflect target domains to avoid spurious transfer.
  • Safety and oversight: In regulated sectors, synthetic training is a complement—not a substitute—for domain data and human review.
  • Scaling variability: γ depends on logical expressiveness and training distribution; estimate per‑model/per‑domain before budgeting.
View Paper Prompt  View All Prompts 

Glossary

  • AIC (Akaike Information Criterion): A model selection metric; lower AIC indicates a better trade-off between goodness of fit and complexity. "the power-law model consistently outperforms an exponential fit across all settings (ΔAIC≥+7.1\Delta\text{AIC} \geq +7.1; see Appendix~\ref{app:fit_comparison})"
  • Antecedent: The “if” part (premises) of an implication in logic. "We only permit conjunctions in the antecedent of if-then statements"
  • Avg@8: An evaluation metric equal to the average accuracy over eight sampled completions per problem. "with each benchmark evaluated using Avg@8, the average accuracy over eight independently sampled completions per problem."
  • Chain-of-thought (CoT): An approach that elicits explicit multi-step reasoning in model outputs. "Under curriculum training, long chain-of-thought behavior emerges at broadly similar stages across depths;"
  • Clip-higher: A training heuristic that adjusts PPO-style clipping to improve stability/efficiency. "our DAPO recipe employs the dynamic sampling and clip-higher strategies from~\citet{yu2025dapo} to improve training efficiency."
  • Consequent: The “then” part (conclusion) of an implication in logic. "Similarly, we only permit disjunctions in the consequent."
  • Conjunction: The logical “and” operator that requires multiple premises to be simultaneously true. "the conjunction operator (i.e., 'and', written {∧\land})"
  • Curriculum training: A strategy that gradually increases task difficulty during training. "and curriculum-based training substantially improves scaling efficiency."
  • DAPO: An RL optimization recipe extending GRPO with additional strategies for efficiency. "Our primary RL algorithm is DAPO~\citep{yu2025dapo}, an extension of Group Relative Policy Optimization (GRPO)~\citep{shao2024deepseekmath}."
  • Disjunction: The logical “or” operator allowing multiple possible conclusions. "the disjunction operator (i.e., 'or', written {∨\lor})"
  • Dynamic sampling: An RL training strategy that adaptively samples outputs/prompts to improve learning efficiency. "our DAPO recipe employs the dynamic sampling and clip-higher strategies from~\citet{yu2025dapo} to improve training efficiency."
  • Expressiveness (logical): The range of constructs a logic can represent (operators, quantifiers, structure). "these gains depend strongly on the logical expressiveness of the training environment, with more expressive settings yielding larger and more compute-efficient transfer."
  • First-order reasoning: Reasoning that includes quantification over entities (e.g., “for all”). "towards more expressive first-order reasoning with conjunction ('and'), disjunction ('or'), negation ('not'), and universal quantification ('for all')."
  • GRPO (Group Relative Policy Optimization): An RL objective that normalizes rewards within sampled groups to compute advantages. "Our primary RL algorithm is DAPO~\citep{yu2025dapo}, an extension of Group Relative Policy Optimization (GRPO)~\citep{shao2024deepseekmath}."
  • GSPO: A sequence-level policy optimization method for RL fine-tuning. "GSPO~\citep{zheng2025gspo}, a recent sequence-level policy optimization variant."
  • Group-normalized advantage: An advantage estimate normalized across a group of sampled completions. "and A^i\hat{A}_i is the group-normalized advantage computed from the scalar completion rewards {Ri}i=1G\{R_i\}_{i=1}^G."
  • Horizon (reasoning horizon): The number of sequential reasoning steps required to solve a task. "once the reasoning horizon is sufficiently large."
  • Implicational propositional logic: A fragment of propositional logic using only the implication operator. "also known as implicational propositional logic or implicational propositional calculus."
  • Log-log space: A coordinate transform applying logarithms to both variables, often used for fitting power laws. "We fit this relationship via ordinary least squares in log-log space."
  • Negation: The logical “not” operator that inverts the truth of a proposition. "the negation operator (i.e., 'not', written {¬\neg})"
  • Ordinary least squares (OLS): A regression method minimizing squared errors between observations and model predictions. "We fit this relationship via ordinary least squares in log-log space."
  • Oracle-solvable: Solvable exactly by a known procedure with guaranteed time complexity. "with each instance oracle-solvable in time linear in the proof size."
  • Out-of-distribution (OOD) generalization: Performance on data that differ from the training distribution. "OOD generalization remains bounded: even the models trained at the largest depths fall to random at $D_{\mathrm{test}/D_{\mathrm{train}\!\approx\!3$."
  • Pass@1: The probability that the model’s first attempt is correct. "our primary measure of training compute, denoted by TT, is the number of RL training steps required to reach a held-out Pass@1 threshold of μ=90%\mu = 90\%."
  • Policy ratio (token-level): In PPO-style methods, the ratio of new to old policy probabilities at each token. "Here, ri,t(θ)r_{i,t}(\theta) is the token-level policy ratio,"
  • Polarity (literal): Whether a literal is positive or negated, affecting its role in proofs. "the model must track not only whether a predicate has been established but also its polarity."
  • Power law: A scaling relationship of the form y ∝ xγ between two quantities. "the RL training compute TT follows a power law with respect to reasoning depth DD"
  • Proof by contradiction: A reasoning principle that derives a conclusion by showing its negation leads to a contradiction. "This may also partly reflect the absence of inference rules such as proof by contradiction."
  • Proof tree: A tree-structured derivation from axioms to a conclusion via inference rules. "Applying the procedure to the BB roots yields one proof tree for each candidate conclusion."
  • RLVR (Reinforcement learning with verifiable rewards): RL where rewards are deterministically checked by a verifier rather than heuristically judged. "RL with verifiable rewards (RLVR) has become a promising paradigm for reasoning-oriented post-training"
  • Scaling exponent: The exponent γ in a power-law relation that governs how quickly cost grows with problem size. "the scaling exponent γ\gamma increases monotonically with logical expressiveness"
  • Scaling-law analysis: Studying systematic relationships between performance and resources (model/data/compute). "Recent work has extended scaling-law analysis to RL post-training,"
  • Self-evolving pipelines: Automated data-generation systems that iteratively create and verify training tasks. "self-evolving pipelines~\citep{zhao2025absolute,huang2025rzero}, reduce the cost of data collection through automatic generation and verification,"
  • Sequence-level policy optimization: RL methods that optimize based on whole-sequence outcomes rather than token-level objectives. "GSPO~\citep{zheng2025gspo}, a recent sequence-level policy optimization variant."
  • Test-time scaling: Improving reasoning by allocating more computation during inference/decoding (e.g., more samples/steps). "Later, test-time scaling emerged to improve reasoning by allocating additional computation during decoding"
  • Universal quantification: A logical construct stating a rule holds for all entities in a domain (e.g., ∀x). "through universal quantification (i.e., 'for all', written {∀\forall})"
  • Verifier: A component that checks whether the model’s answer matches the ground truth under a specified format. "During evaluation, the verifier extracts the predicted answer from this span and compares it with the ground-truth answer via exact match."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 17 tweets with 94 likes about this paper.

Sign Up for Free