GPG: Generalized Policy Gradient Theorem for Transformer-based Policies

Abstract: We present the Generalized Policy Gradient (GPG) Theorem, specifically designed for Transformer-based policies. Notably, we demonstrate that both standard Policy Gradient Theorem and GRPO emerge as special cases within our GPG framework. Furthermore, we explore its practical applications in training LLMs, offering new insights into efficient policy optimization.