Reducing Dependence on Benchmark Coverage for Rater Supervision

Develop methods to learn capability-specific raters for SkillRater using broader capability taxonomies or weaker supervision so that capabilities not represented in the validation benchmarks can be targeted.

Background

The current SkillRater raters are trained using capability-specific validation benchmarks. This supervision limits which capabilities can be targeted: if a capability lacks a corresponding benchmark, it cannot be directly optimized.

The authors identify reducing dependence on benchmark coverage as an open direction, proposing expansion to broader capability taxonomies or leveraging weaker supervision signals to cover underrepresented capabilities.