Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization

Abstract: Scientific reasoning is an increasingly important capability of LLMs, yet improving the robustness and efficiency of training such reasoning remains a key open challenge. We study this problem in instruction-based molecular optimization, where answer-only supervised fine-tuning (SFT) collapses multi-step reasoning and reinforcement learning with verifiable rewards (RLVR) suffers from sparse feedback. Reference-guided Policy Optimization mitigates both by anchoring policy updates to dataset-provided references, but its effectiveness is tightly coupled to reference quality: weak or misaligned references impose a performance ceiling. To overcome this ceiling, we propose active reasoning, a paradigm in which the policy actively decides, on a per-instance basis, when to imitate a reference and when to reinforce its own discoveries, while continuously upgrading what it imitates. We instantiate this paradigm as Active Group Relative Policy Optimization (Active-GRPO), realized through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when the reference still outperforms the policy's own candidates, and shifts to self-improvement via reinforcement learning once the policy has generated molecules that surpass the reference. The latter continuously upgrades the reference itself by replacing it with the best policy-generated candidate discovered so far, progressively raising the imitation target and ensuring that reference guidance remains informative-rather than restrictive-throughout training. Across TOMG-Bench MOLOPT, Active-GRPO improves average SRxSim from 0.0959 for GRPO and 0.1665 for RePO to 0.1773 under matched three-seed evaluation, with statistically significant gains on LogP, MR, and QED.

• The paper introduces a novel dynamic framework that transitions from imitation to reinforcement learning for improved molecular optimization performance.
• It integrates active imitate-reinforce and active referencing mechanisms to adaptively upgrade reference quality and guide learning.
• Empirical results show significant gains in SR×Sim metrics and reduced no-op failure rates across molecular property optimization tasks.

Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization

Motivation and Problem Setting

Designing LLMs capable of robust instruction-conditioned molecular optimization is central to accelerating scientific discovery, drug design, and materials engineering. Conventional methods—namely supervised fine-tuning (SFT) and RL with verifiable rewards (RLVR)—face salient limitations: SFT collapses complex reasoning into trivial map-memorization, while RLVR suffers from reward sparsity due to stringent chemical validity and similarity constraints. Reference-guided Policy Optimization (RePO) bridges these to a degree by introducing answer-level imitation on reference molecules, but critically, its efficacy is gated by the static quality of these references. When dataset-provided references are stale or suboptimal, the policy is actively prevented from surpassing the baseline, resulting in a stringent performance ceiling.

Active-GRPO directly addresses this bottleneck by formulating molecular optimization as an active reasoning problem: policies dynamically arbitrate between imitation and self-improvement, simultaneously upgrading "what" and "when" to imitate as performance evolves. This brings forth a self-reinforcing, policy-conditional regime—anchoring learning to human references only so long as they are beneficial and transitioning to policy-discovered exemplars once the policy has exceeded them.

Figure 1: Conceptual motivation for Active-GRPO: RePO's static reference can become suboptimal as the policy improves, whereas Active-GRPO transitions adaptively from reference imitation to policy improvement.

Methodology: Active-GRPO Mechanisms

Active-GRPO operationalizes active reasoning with two intertwined algorithmic components: active imitate-reinforce and active referencing.

• Active Imitate-Reinforce: For each example, the guidance weight $\beta_{\text{guide}}(c)$ is computed based on the gap between the best policy sample’s reward and that of the current reference. When the reference is superior, imitation (large $\beta$) dominates the gradient; as the policy surpasses the reference, $\beta$ decreases, effecting a smooth shift toward pure RL.
• Active Referencing: For each context, a memory bank maintains a top-$K$ set of policy-generated molecules that have outperformed the static reference. Once a policy-generated candidate surpasses the reference (by a margin $\delta$ and satisfying admissibility predicates, such as minimum similarity), it is promoted to the reference set. Guidance loss is always calculated with respect to the currently best candidate in the bank (dataset or promoted).

The combined objective is

$$\mathcal{L}_{\text{Active-GRPO}} = \mathcal{L}_{\mathrm{RL}} + \frac{1}{B}\sum_{i=1}^B \beta_{\text{guide}}(c_i) \mathcal{L}_{\text{guide}}^{(i)}(m^*(c_i))$$

where the $\mathcal{L}_{\mathrm{RL}}$ is the GRPO-style RL loss and the guidance target $m^*(c_i)$ is dynamically chosen.

Figure 2: Overview of Active-GRPO, illustrating dynamic guidance weighting and memory-bank-based active referencing for each instance.

Experimental Results

Active-GRPO is benchmarked against previous SFT/RLVR approaches, iterative self-distillation, and non-adaptive strong reference baselines on TOMG-Bench MolOpt (LogP, MR, QED). All experiments use matched architectures and budgets to ensure controlled comparison. The primary evaluation metric is SR$\times$Sim, capturing both property success and scaffold preservation.

Numerical Performance

Active-GRPO shows consistent improvement over all baselines:

• SR$\times$Sim increases from 0.1665 (RePO) to 0.1773 (Active-GRPO) averaged across tasks.
• For LogP, the gain relative to RePO (+0.0148) grows monotonically with the unused optimization headroom—verifying the central hypothesis that adaptive referencing is most beneficial as static references become uninformative.
• Ablations confirm that neither dynamic guidance nor memory bank alone suffices; their combination yields the observed gains.

  Figure 3: SR$\beta$0Sim increments with optimization headroom, confirming that Active-GRPO most assists when references lag policy capability.

Training Dynamics

Analysis of training curves shows:

• Guidance weights shift from high to low as training progresses, signaling transition from imitation to RL.
• The memory bank is steadily filled with policy-promoted molecules, and a significant fraction of examples use self-promoted rather than dataset references mid-to-late training.

  Figure 4: Training curves for reward, RL loss, average guidance weight, and self-distillation activity of Active-GRPO.

Qualitative Analysis

Case studies indicate that Active-GRPO reliably makes targeted structural edits (such as hydrophobic substitutions for LogP or ring simplifications for MR) that satisfy requested properties while maintaining scaffold integrity. In contrast, RePO often copies the source molecule unchanged or makes edits that degrade the desired property—both modes reflecting the liabilities of static-reference anchoring.

Figure 5: Representative edits—Active-GRPO achieves property improvements with minimal unnecessary modification, while RePO often performs no-op or incorrect edits.

Moreover, Active-GRPO substantially reduces the no-op failure rate (outputs nearly identical to the source but failing the property objective), indicating that lower average similarity for Active-GRPO reflects actual productive editing rather than structural destruction.

Figure 6: Active-GRPO reduces no-op failures (high similarity + failure to solve the property objective) compared to RePO across all subtasks.

Implications, Limitations, and Future Directions

Theoretical implications of Active-GRPO's success point to the necessity of dynamic supervision targets in domains where both expert and environmental optimality are partial and non-stationary. Practically, integrating active referencing and adaptive imitation can drive effective LLM-based reasoners beyond any external data-cap ceiling—a critical property in low-resource or rapidly evolving scientific domains.

The transferability of this approach to broader de novo discovery, where high-quality references may be sparse or absent, suggests promising future research in integrating curriculum construction, exploration strategies, or even fully policy-driven bootstrapping. Conversely, extension to multi-objective regimes with conflicting constraints and new reward landscapes will require further analysis of bank management, promotion criteria, and guidance adaptation.

Conclusion

Active-GRPO establishes a principled framework for dynamic reference-guided policy optimization, combining context-dependent imitation strength and policy-conditional reference promotion. Empirical results demonstrate statistically robust improvements over static-reference baselines across all molecular optimization metrics. The method’s efficacy is maximized precisely where conventional approaches fail—when dataset supervision is insufficient and the agent must outpace its dataset. This advances the state-of-the-art in LLM-driven scientific reasoning under constrained optimization and sets the stage for future active self-improving RL frameworks in AI-driven discovery.