SDPO for alignment in open-ended or continuous‑reward settings

Determine whether Self‑Distillation Policy Optimization (SDPO), which distills a feedback‑conditioned self‑teacher into the policy, improves alignment in open‑ended text generation and in continuous‑reward tasks that lack a ground‑truth verifier, by empirically evaluating its retrospection‑based credit assignment in such settings.

Background

The paper introduces Reinforcement Learning with Rich Feedback (RLRF) and proposes Self‑Distillation Policy Optimization (SDPO), which uses a feedback‑conditioned self‑teacher to provide dense, logit‑level credit assignment. Experiments primarily focus on verifiable domains such as code generation, where rich environment feedback (e.g., runtime errors, unit tests) is available.

The authors note that many real‑world tasks provide textual feedback without a ground‑truth verifier. Extending SDPO beyond verifiable settings raises the question of whether the retrospection mechanism can improve model alignment when rewards are not purely binary or verifiable, such as in open‑ended text generation or tasks with continuous rewards.