DecompRL: Solving Harder Problems by Learning Modular Code Generation

Abstract: How can LLMs solve problems they currently cannot? Repeated sampling scales test-time compute but GPU cost grows linearly with attempts, while reinforcement learning (RL) with verifiable rewards improves single-attempt accuracy at the expense of sample diversity. Both strategies ultimately fail when the base policy has near-zero probability of producing a correct solution: no amount of sampling or gradient signal can overcome a search space that is simply too large. We take a different approach: rather than sampling harder, we make the task easier by decomposing problems into smaller, independently solvable sub-functions whose implementations can be recombined. Since off-the-shelf models are not trained for this modular generation, we introduce DecompRL, an RL algorithm that explicitly learns to decompose and implement hierarchical code structures. Recombining $k$ implementations of $n$ modules yields up to $k^{n}$ candidate solutions, shifting the bottleneck from GPU inference to cheap CPU evaluation and cutting GPU token cost by $\sim$50$\times$. On LiveCodeBench and CodeContests (Qwen~2.5~7B, Code World Model~32B), DecompRL outperforms standard and diversity-optimized RL baselines beyond $10^5$ tokens per problem, solving problems that standard generation cannot reach.

• The paper introduces DecompRL, a modular RL framework that decomposes challenging code tasks into independent sub-functions, improving scaling efficiency.
• It employs two policies—a decomposition policy for modular planning and an implementation policy for generating diverse sub-code candidates—to enhance exploration.
• The approach achieves up to 35% success on hard benchmarks with 50× fewer tokens, underscoring significant improvements in efficiency and solution diversity.

DecompRL: Modular Hierarchical Reinforcement Learning for Scalable Code Generation

Introduction and Motivation

DecompRL presents a reinforcement learning (RL) framework aimed at addressing the limitations of standard LLM-driven code generation in verifiable domains, such as competitive programming. Existing paradigms scale solution accuracy either through repeated sampling (pass@$k$) at linearly increasing GPU cost or through RL finetuning that improves pass@1 but at the expense of sample diversity and diminishing returns for high $k$. Both approaches falter when a task's intrinsic complexity yields nearly zero base policy solve rates, as neither brute-force sampling nor gradient-based optimization over the monolithic code search space can make substantive progress.

DecompRL's central contribution is to reparameterize the generation problem via explicit learned modular decomposition, coupled with independently optimized implementation policies for sub-functions. This hierarchical agent splits challenging problems into $n$ components, generates $k$ implementations for each, and recombines them, yielding up to $k^n$ unique candidates per problem at only $O(nk)$ inference cost. The critical innovation is shifting the search bottleneck from expensive GPU-based generation to cheap CPU-bound verification (Figure 1), enabling unprecedented scaling in the effective solution search space.

Figure 1: Decomp RL solves harder problems than existing RL methods. Hierarchical training learns both decomposition and implementation policies, yielding higher solve rates on the hardest LiveCodeBench problems.

Hierarchical Inference and Policy Architecture

The DecompRL agent factorizes code generation into two policies:

• The decomposition policy $\pi(D)$ maps the problem into a set of function signatures and docstrings, serving as the modular plan.
• The implementation policy $\pi(I_i|D)$ instantiates candidate code for each sub-function independently, conditioned on the decomposition.

This departs from the strictly autoregressive, joint generation of all function bodies and instead assumes conditional independence between implementations given the plan. The modularity enables exponential recombinatorial scaling: for $n$ modules and $k$ implementations each, the agent can create and evaluate $k$0 complete code solutions for only $k$1 LM forwards (Figure 2).

Figure 2: Hierarchical inference enables exponential scaling of candidate solutions for a fixed number of model calls by recombining partial implementations.

Crucially, this architecture presupposes domains with cheap, reliable verification (e.g., unit tests or theorem proving) and a naturally modular solution structure.

Reinforcement Learning and Reward Aggregation

DecompRL's learning objective explicitly targets high-diversity, composable search. Standard policy gradients are adapted to exploit the combinatorial reward matrix from modular recombination. This enables lower-variance gradient estimation, as the reward for each sub-policy action can be computed as an average (or more generally, a softmaxed aggregate) over an exponentially large pool of sampled candidates, rather than over sequence-aligned joint samples.

Reward aggregation leverages the logmeanexp ($k$2) operator with tunable temperature $k$3, interpolating between maximizing for best-case (pass@k) versus mean-case performance. This softly addresses the usual tradeoff between exploitation (high pass@1) and exploration (maintaining diversity as measured by high pass@k). Further, leveraging a leave-one-out baseline for advantage calculation in the policy gradient reduces variance and aligns credit assignment more closely with the contribution of each action to global success (Figure 3 and Figure 4).

Figure 3: Nonlinear reward transformations induced by logmeanexp and pass@k objectives; lower $k$4 promotes exploration by prioritizing rare successes.

Figure 4: Ablation study on DecompRL's objective: removing individual components (logmeanexp, sequential training, leave-one-out) leads to entropy or reward collapse.

Sequential multi-agent training alternates optimization between decomposition and implementation policies, mitigating nonstationarity and overfitting that would arise with naïve joint training.

Empirical Evaluation

Datasets and Training Setup

Experiments are conducted on the competitive programming datasets CodeContests, TACO, and LiveCodeBench. Architectures evaluated include Qwen 2.5 7B, Llama 3.1 8B Instruct, and Code World Model 32B, with distributed PPO-based RL and up to $k$5 decompositions, each with $k$6 candidate implementations per function.

Baselines

DecompRL is compared against state-of-the-art RL approaches focused on pass@1 (GRPO), pass@k training, Soft Policy Optimization (SPO), and their own logmeanexp-based baseline. All approaches use comparable computational budgets.

Main Results

DecompRL demonstrates clear empirical superiority in the high-$k$7 evaluation regime. For instance, on the hardest LiveCodeBench subset, DecompRL achieves up to $k$8 success for the hardest problems, outperforming all baselines at budgets above $k$9 tokens per problem. Notably, DecompRL maintains non-saturating gains as the number of candidate code evaluations increases, unlike standard RL methods which plateau (Figure 1).

The method combines strong pass@k scaling with substantial GPU savings: for $n$0 decompositions of up to $n$1 functions and $n$2 implementations each, DecompRL can evaluate $n$3 candidates while generating only $n$4k tokens—approximately $n$5 less than the baseline's equivalent—shifting nearly all wall-clock cost to CPU verification. These compute efficiencies persist even as the scaling regime is pushed to millions of candidates per problem.

Hierarchical models not only solve strictly “hard” instances unreachable by any baseline but also uncover new correct solutions, indicating enhanced exploration and robustness.

Diversity and Ablation

Hierarchical recombination leads to a meaningful increase in output diversity, as reflected in the extended, non-plateauing success curve with increased evaluations (Figure 2).

Ablation studies unambiguously show that reward aggregation via $n$6, sequential multi-agent training, and the leave-one-out baseline are all necessary for stable, non-collapsing training (Figure 4), preventing overfitting ("reward hacking") and maintaining reasonable decomposition granularity (as measured by function count).

Limitations

DecompRL imposes a "format tax": in low-budget or easy-problem regimes, hierarchical inference without RL underperforms standard inference due to the precision loss of conditional independence and off-policy distribution mismatch (i.e., most training data are monolithic, not modular). Furthermore, the "size collapse" phenomenon, wherein the decomposition granularity decays under RL pressure, threatens to regress toward standard generation, which must be guarded against using reward shaping and alternating policy optimization. The method is thus best suited to tasks where monolithic solutions are intractable and evaluation is cheap relative to generation.

Implications and Future Directions

Practically, DecompRL enables efficient, large-scale program search in situations where GPU compute is the hard bottleneck, such as competitive programming, automated AI-system design, and hierarchical scientific modeling. The framework’s modular generation pipeline introduces new axes of control and interpretability (e.g., function-level debuggability), and provides natural hooks for future research in modular, self-improving systems.

From a theoretical perspective, DecompRL directly addresses the hard-exploration problem endemic to LLM-based RL, and points to the clear advantage of modular, factorized policies with recombinatorial credit assignment. The approach can be viewed as a synergistic “explorer” agent for standard distillation pipelines: DecompRL can supply high-diversity, hard-to-discover behaviors for subsequent pass@1 policy distillation, potentially generalizing to non-coding domains where modular subgoal structure and automatic verification are available.

Further scale-up and systematic investigation of modularity-induced exploration in broader RLHF and Imitation Learning frameworks are natural next steps. In the present context, larger and more parallel CPU resources for verification could unlock further search gains.

Conclusion

DecompRL constitutes a principled solution to exploration in LLM-based code generation, leveraging learned modular decomposition and aggressive inference-time recombination under a multiturn RL objective. The algorithm achieves strong gains for hard, verifiable tasks at a fraction of the inference token cost associated with traditional RL or repeated sampling approaches. Its framework is theoretically sound, practically effective for large token budgets, and opens new avenues for modular, interpretable, and highly extensible RL with LLMs.

Figure 5: Cross-evaluation of independently trained decomposition and implementation models underlines the benefit of joint, alternating optimization on CodeContests.

Figure 6: Policy entropies during training reveal exploitation-exploration dynamics: sequential/alternating policy updates stall entropy collapse.

Figure 7: LiveCodeBench baseline performances show that only DecompRL scaling escapes saturation and continues improving with increased candidate evaluations.

• Decugis, J., et al. “DecompRL: Solving Harder Problems by Learning Modular Code Generation” (2607.02390).