Sufficiency of soft verification at larger scales

Determine whether soft verification, defined as selecting synthetic coding trajectories by line-level patch recall without executing unit tests, remains sufficient to improve coding agent performance when scaling to larger base models and greater training data volumes, or whether unit test–based hard verification becomes necessary for further gains.

Background

The paper introduces Soft Verified Generation (SVG), which uses line-level patch recall to select training trajectories instead of unit test execution. Ablations show little difference among verification thresholds at the tested scales, suggesting verification may not be essential for early gains.

In the Limitations, the authors note they were unable to test whether this conclusion holds at larger scales and speculate that hard verification might become necessary as models and datasets grow.