CWRPO: Canvas Workflow Relative Policy Optimization

• The paper introduces CWRPO, a reinforcement learning method that enhances workflow orchestration with token-level control and curriculum-based rewards.
• It employs a structured canvas MDP with editing actions and a modified PPO surrogate objective to manage multi-turn, agentic workflows.
• Empirical evaluations show significant accuracy gains across benchmarks by mitigating degenerate policies and ensuring diverse, scaffolded trajectories.

Canvas Workflow Relative Policy Optimization (CWRPO) is a reinforcement learning (RL) method introduced for automating agentic workflow orchestration in executable canvas environments. Developed within the FlowSteer framework, CWRPO addresses challenges including sparse reward signals, high manual orchestration costs, and susceptibility to degenerate “shortcut” behaviors by implementing a structurally constrained, curriculum-based reward system and a Relative Policy Optimization strategy with token-level control and regularization (Zhang et al., 2 Feb 2026).

1. Canvas MDP Formulation

CWRPO operates in the context of a Markov Decision Process (MDP) tailored to multi-turn, agentic workflow construction over an executable canvas.

• State Space ($S$): Each state at step $t$, denoted $H_t \in S$, aggregates the complete interaction history:

$$H_t = [q \oplus d^{lib} \oplus a^{tmpl} \oplus (a^{think}_1, a_1, o^{exec}_1),..., (a^{think}_t, a_t, o^{exec}_t)]$$

where $q$ is the task, $d^{lib}$ encodes available operators, $a^{tmpl}$ specifies prompt templates, $a^{think}_\tau$ is the agent's internal thought, $a_\tau$ is the editing action, and $o^{exec}_\tau$ the feedback.

• Action Space ($A$): Editing actions $a_t = (\alpha_t, a^{out}_t)$ comprise a type $\alpha_t \in$ {add, delete, modify, set_prompt, parallel, conditional, loop, finish} and pertinent content $a^{out}_t$.
• Transition Dynamics: After $H_{t-1}$ and $a_t$, the canvas updates $G_t = \text{Update}(G_{t-1}, a_t)$ and samples feedback $o^{exec}_t \sim C_{exec}(\cdot|G_{t-1}, a_t)$. The interaction ends if $\alpha_t =$ finish or $t = T_{max}$.
• Policy Factorization: The policy $\pi_\theta(a_t^{think}, a_t | H_{t-1})$ is factorized:

$$\pi_\theta(a_t^{think} | H_{t-1}) \cdot \pi_\theta(\alpha_t | H_{t-1}, a_t^{think}) \cdot \pi_\theta(a_t^{out} | H_{t-1}, a_t^{think}, \alpha_t)$$

This formalization underpins CWRPO’s tailored optimization and reward gating mechanisms.

2. Objective Function and Token-Level Optimization

CWRPO directly modifies the Proximal Policy Optimization (PPO) objective to suit the canvas domain, leveraging trajectory-level normalization and token-level control:

• Batch Trajectories: A batch $\{\tau_i\}_{i=1}^N$ is sampled using $\pi_{\theta_{old}}$, each represented as tokens $(w_1,...,w_{|\tau_i|})$. Environment-generated tokens are masked ($\text{mask}_t^{(i)} \in \{0,1\}$).
• Token Importance Ratio:

$$\rho_\theta(w_t^{(i)}) = \frac{\pi_\theta(w_t^{(i)} | w_{<t}^{(i)})}{\pi_{\theta_{old}}(w_t^{(i)} | w_{<t}^{(i)})}$$

• Group-Relative Advantage:

$$\hat{A}_i = \frac{R(\tau_i) - \mu_{src}}{\sigma_{src} + \epsilon}$$

where $\mu_{src}, \sigma_{src}$ are per-source mean and variance.

• Surrogate Objective:

$$J_{CWRPO}(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{old}}} \Bigg[ \frac{1}{N}\sum_{i=1}^N \frac{1}{|\tau_i|_{mask}} \sum_{t=1}^{|\tau_i|} \text{mask}_t^{(i)} \min\left( \rho_\theta(w_t^{(i)})\hat{A}_i, \text{clip}(\rho_\theta(w_t^{(i)}),1-\epsilon,1+\epsilon)\hat{A}_i \right) \Bigg] - \beta D_{KL}(\pi_\theta \| \pi_{ref})$$

with KL regularization towards a reference policy $\pi_{ref}$ and PPO-style clipping range $\epsilon$.

Token-level masking ensures unbiased gradients, contributing only policy-generated tokens to the surrogate, mitigating variance from environment dynamics.

3. Structural Reward Engineering and Conditional Release

CWRPO employs a two-stage, conditional-release reward system to enforce workflow diversity prior to answer optimization.

• Structural Diversity Score ($R_{div}(\tau)$):

$$R_{div}(\tau) = \min(1.0, R_{chk} + R_{fmt} + R_{op} + R_{ctrl})$$

where: - $R_{chk}$: 0.25 if at least one verification operator (Test/Review/Verify) is present. - $R_{fmt}$: 0.25 if “Format” is included as the final step. - $R_{op}$: 0.25 if at least three distinct operators are used. - $R_{ctrl}$: 0.25 if any control structure (parallel/conditional/loop) is present.

• Answer Reward ($R_{ans}(\tau)$): Assessed as task-appropriate correctness, typically via $[0,1]$ pass-rate or a specialized evaluation.
• Total Reward (Conditional Release):

$$R(\tau) = -1.0 + R_{div}(\tau) + \mathbf{1}\{R_{div}(\tau) = 1.0\} R_{ans}(\tau)$$

Only skeleton-complete trajectories with $R_{div}=1$ receive answer rewards; others are restricted to negatively shifted, diversity-based feedback. This design suppresses degenerate or shortcut behaviors in RL.

4. Algorithmic Implementation and Distinctions from PPO

CWRPO’s update schedule introduces differentiating features relative to standard PPO, codified in the following workflow:

• Sampling: $N$ trajectories are generated with the current policy.
• Reward Computation: Both structural diversity and, upon satisfaction, answer correctness rewards are computed.
• Advantage Normalization: Rewards are normalized via group statistics for stabilization.
• Token Clipped Objective: The surrogate update aggregates over only policy-generated tokens, clipped by $\epsilon$.
• KL Regularization: A KL penalty is imposed against the reference (e.g., initial SFT) policy.
• Parameter Updates: Gradients are computed to optimize the summed objective; $\theta_{old}$ is refreshed every epoch.

Key Implementation Elements	Value/Setting	Description
PPO clipping ($\epsilon$)	0.20	Safe trust region scaling
KL penalty ($\beta$)	0.005	Regularization strength
Advantage norm safety ($\delta$)	$10^{-8}$	Stability in denominator
Batch size ($N$)	36	Trajectories per update
Max turns ($T_{max}$)	20	Horizon per interaction
Operator-specific weights	0.25 each, capped at 1.0	Structural reward configuration

LoRA fine-tuning on compact policy models (e.g., Qwen3-8B) with bfloat16 and gradient checkpointing is utilized. Explicit canvas-level constraints enforce minimal operator usage and prevent premature “finish” calls.

5. Theoretical Properties and Training Dynamics

CWRPO is characterized by structural-constraint separation and curriculum-based policy shaping:

• Reward Sign Separation: Trajectories lacking the structural skeleton ($R_{div}<1$) are capped below zero, while only skeleton-complete ones ($R_{div}=1$) admit potential positive reward.
• Curriculum Induction: Early training is dominated by learning feasible (diverse) workflow skeletons, transitioning to answer optimization as feasible probability $p_\theta = \Pr(R_{div}=1)$ approaches unity.
• Monotonicity: By importance ratio clipping and KL regularization, per-update improvements are bounded analogously to PPO; token-level masking assures unbiased stochastic gradient estimates.

Under constraints of bounded rewards and hyperparameter limits ($\alpha, \epsilon, \beta$ sufficiently small), local improvement guarantees parallel those in PPO.

6. Benchmark Performance and Empirical Validation

CWRPO’s efficacy is substantiated across twelve datasets, spanning both in-distribution (IID) and out-of-distribution (OOD) tasks, in comparison to leading orchestration baselines:

• In-Distribution Results: Notable outcomes include 96.09% on GSM8K (vs. 93.75% for the best agent-RL baseline), 81.25% on MATH (vs 74.22%), >10 point improvements on HotPotQA and SQuAD-v2 EM/F1, and +10–20% Pass@1 lift on MBPP and HumanEval.
• OOD Generalization: CWRPO retains top accuracy across TriviaQA, NaturalQ, MathQA, AIME, APPS, DS-1000, with +6–16 point gains over strongest baselines.
• Ablation Analyses: Removing structural gating yields shortcut policies; omitting masking or KL penalty destabilizes optimization; the full combination of multi-turn canvas, answer masking, and CWRPO is necessary for observed performance.

Direct RL comparisons indicate that CWRPO outperforms DAPO and GRPO across math, QA, and code orchestration settings under identical evaluation regimes (Zhang et al., 2 Feb 2026).

7. Context and Implications

CWRPO, by incorporating diversity-constrained, conditional-release reward structures and token-level optimization atop PPO, demonstrates substantial gains in stability, diversity, and performance for agentic workflow orchestration. Its explicit separation of structural and correctness phases, combined with targeted regularization and unbiased gradient computation, marks a significant methodological advancement for RL in complex, long-horizon, multi-turn domains. This approach underlines the relevance of curriculum design, reward gating, and tight stochastic control in automated workflow construction, and sets a new empirical benchmark for workflow orchestration in both familiar and unseen environments (Zhang et al., 2 Feb 2026).