STRL: Sketch-Thinking Reinforcement Learning

• Sketch-Thinking Reinforcement Learning is a paradigm that uses abstract sketches to impose compositional structure and guide policy optimization in RL.
• It employs diverse representations like symbolic subtasks, visual strokes, and multimodal tokens to break down complex tasks and improve credit assignment.
• STRL demonstrates significant performance gains in applications such as image retrieval, robotics, and multimodal reasoning through hierarchical policies and refined reward shaping.

Sketch-Thinking Reinforcement Learning (STRL) encompasses a family of reinforcement learning paradigms in which “sketches”—partial, abstracted, or symbolic representations of task structure, intermediate visualizations, or stepwise reasoning traces—are explicitly manipulated, generated, or utilized to guide policy optimization, improve credit assignment, or enhance interpretability. STRL addresses domains ranging from image retrieval and fine-grained abstraction, to multimodal reasoning and long-horizon robotics, leveraging the sketch modality to foster efficient, robust, and human-aligned solutions.

1. Foundational Concepts of Sketch-Thinking in RL

STRL formalizes sketches as intermediary artifacts that capture critical substructure: they may be symbolic (ordered subtask lists in policy sketches), structural (dynamic visual markers in reasoning over images), or physical (robotic stroke trajectories). Unlike standard RL, which acts directly in raw state-action space, STRL leverages sketches to impose compositional or hierarchical structure, bias training toward means–end abstraction, or externalize internal “thought” processes for downstream utility.

In policy sketches (Andreas et al., 2016), a sketch $K_\tau = (b_1, b_2, ..., b_N)$ specifies a sequence over a vocabulary $B$ of subtasks but omits low-level implementations, guiding RL to discover interchangeable primitives. In image understanding or robotic drawing, a sketch is a sequence of visual strokes or marks; agents learn policies that generate or select optimal sketches under task constraints (Bhunia et al., 2020, Bhunia et al., 2022, Muhammad et al., 2018). In multimodal reasoning, sketches become interleaved CoT traces, explicit image annotations, or visual actions interleaved with language tokens (Zheng et al., 20 May 2025, Zhang et al., 6 Jan 2026, Huang et al., 9 Jan 2026).

2. RL Problem Formalizations and Action Spaces

STRL encompasses a range of MDP and POMDP formulations, each rooted in the semantics of the underlying sketch artifact:

• Stroke Selection and Abstraction: The agent operates over a vectorized sketch as a sequence $(s_1, ..., s_K)$, with per-stroke binary actions $a_i \in \{\text{select},\text{ignore}\}$ yielding a subset sketch $\bar S_V$ (Bhunia et al., 2022). In deep abstraction, actions decide for each segment whether to keep or skip, trading brevity against recognizability (Muhammad et al., 2018).
• Sequential Reasoning and Multimodal Token Emission: Policy acts in an interleaved sequence space of text/image/tool actions: at each step, chooses a text token or a visual tool call (e.g., emitting a bounding box or sketch marker), progressively building up a reasoning trajectory $s_t = \{(X_0,I_0),...,(X_t,I_t)\}$ (Zheng et al., 20 May 2025, Huang et al., 9 Jan 2026).
• Hierarchical/Modular Policies: Tasks are represented as sketches over subtasks, each realized by a neural subpolicy $\pi_b(s;\theta_b)$; their concatenation implements the full task policy (Andreas et al., 2016).
• Robot Drawing/Manipulation: In visual-motor tasks, sketches may be 2D or 3D trajectories, drawing primitives, or sequences of pen commands. RL agents operate in either full trajectory spaces (TD3, DDPG) or over discrete stroke spaces as in DQN/SAC (Fernandez-Fernandez et al., 2024, Lee et al., 2022, Tan et al., 4 Jan 2026, Yu et al., 14 Mar 2025).

3. Reward Shaping, Credit Assignment, and Training Objectives

Reward design in STRL is tightly coupled to the sketch interface:

• Downstream-Driven Rewards: Sketches are evaluated by their contribution to the main task—e.g., stroke subset selection is rewarded by $1/\text{rank}$ in an FG-SBIR gallery and negative triplet loss (Bhunia et al., 2022).
• Fine-Grained Credit Assignment: In chart reasoning, rewards are attributed stepwise by a reward model (FinePRM) that scores each marker; these scores redistribute trajectory-level reward, breaking up global credit into token-level signals for precise error assignment (Huang et al., 9 Jan 2026).
• Conciseness and Sketch-Style: For efficient reasoning, rewards combine accuracy, output format adherence, and a sketch-style signal predicted by a separately trained reward model (SketchJudge) (Zhang et al., 6 Jan 2026).
• Imitation-Aided RL: Sketch trajectories or stroke demonstrations bootstrap policy learning—rewarding pixel similarity improvements, semantic recognizability, or proximity to trajectory distributions (Fernandez-Fernandez et al., 2024, Zhou et al., 2018, Yu et al., 14 Mar 2025, Lee et al., 2022).
• Sparse and Structural Rewards: In modular multitask RL, reward is only given upon reaching the final or subgoal state; no reward shaping over subtasks is required (Andreas et al., 2016, Aichmüller et al., 2024).

Optimization is done via PPO, actor–critic, SAC, DQN, or TD3, depending on domain, often with custom surrogate losses or advantage normalization to stabilize training and balance reward components.

4. Architectures and Algorithmic Strategies

Architectural choices in STRL are dictated by the sketch’s structure and the demands of the downstream task:

• Hierarchical and Relational Encoders: Stroke selection leverages a local LSTM for individual strokes, a global relational LSTM for temporal context, fused via residual addition and layer normalization (Bhunia et al., 2022). Relational GNNs are employed for sketch decompositions in symbolic planning, encoding objects and pairwise relations (Aichmüller et al., 2024).
• Two-Stream Networks: Combined global-local context (canvas and focus patch) networks are standard for drawing agents, fusing coarse and fine features (Zhou et al., 2018, Fernandez-Fernandez et al., 2024).
• Multimodal Transformers: In reasoning over images/text, transformers with interleaved token streams handle vision, language, and sketch tokens, aligning modalities through attention (Tan et al., 4 Jan 2026, Zheng et al., 20 May 2025).
• Policy Sketch Modules: Modular subpolicies share parameters across tasks, instantiated per sketch symbol, enabling task transfer and explicit decomposition (Andreas et al., 2016).
• Reward Models for Style: Auxiliary networks (FinePRM, SketchJudge) score stepwise behavior (e.g., action quality or sketch-style reasoning) and inject these gradients during RL (Huang et al., 9 Jan 2026, Zhang et al., 6 Jan 2026).

A wide array of training techniques are used, including supervised pretraining on sketch/demonstration data, curriculum learning over sketch length or difficulty, and hierarchical imitation–reinforcement curricula (Yu et al., 14 Mar 2025, Lee et al., 2022).

5. Empirical Results and Quantitative Impact

STRL frameworks deliver significant empirical gains across visual retrieval, reasoning, abstraction, and control:

Application Area	STRL Variant	Reported Gain	Reference
FG-SBIR	Stroke subset RL	+8–10% top-1/top-5 accuracy	(Bhunia et al., 2022)
Sketch abstraction	RL stroke removal	Retain 80–90% recognizability	(Muhammad et al., 2018)
Chart reasoning	FinePO + sketching	+7.2% accuracy vs. baseline	(Huang et al., 9 Jan 2026)
Multimodal CoT	Sketch RL (GRPO)	65% token cost reduction	(Zhang et al., 6 Jan 2026)
On-the-fly FG-SBIR	Sequential PPO	4–6% m@A/m@B lift	(Bhunia et al., 2020)
Human-like robotic	DQN, HRL	Near–perfect sketch fidelity	(Fernandez-Fernandez et al., 2024Lee et al., 2022)
Robotic manipulation	Sketch-to-Skill RL	96% of teleop performance, +170% vs. pure RL	(Yu et al., 14 Mar 2025)

Ablation experiments consistently show that omitting the sketch module (stroke selector, reward model, or explicit sketch action) degrades performance and generalization. Plug-and-play uses of trained STRL modules include sketch data cleaning, data augmentation via stochastic partial re-sketching, and early stopping in sketch-based retrieval.

6. Interpretability, Generalization, and Human Alignment

STRL frameworks provide structured, interpretable outputs. In modular multitask RL, each subpolicy is uniquely mapped to a subtask, rendering the learned policy library directly reusable and human-inspectable (Andreas et al., 2016). Visual-sketching agents produce explicit marks, boxes, or arrows on images, permitting human-in-the-loop correction and causal explanation (Tan et al., 4 Jan 2026). In sketch abstraction, reward and policy saliency maps highlight the most critical details per category, aligning with human perceptual abstraction (Muhammad et al., 2018). Stepwise reasoning agents (DeepEyes, SketchVL, SketchThinker-R1) manifest “sketch-like” reasoning, prioritizing salient cues, and focusing attention through dynamic visual actions, mirroring human cognitive efficiency and interpretability (Zheng et al., 20 May 2025, Huang et al., 9 Jan 2026, Zhang et al., 6 Jan 2026).

Zero-shot and rapid adaptation results in modular/multitask settings indicate robust generalization. STRL architectures excel at compositional transfer: new combinations of known primitives (policy sketch symbols, visual tools) solve novel tasks without retraining (Andreas et al., 2016, Aichmüller et al., 2024).

7. Extensions, Limitations, and Future Directions

Ongoing extensions of STRL target richer forms of sketching (e.g., encoded 3D structures; multi-stage or temporally dynamic sketches), direct language-to-sketch or sketch-to-action learning (Tan et al., 4 Jan 2026, Yu et al., 14 Mar 2025), and the fusion of sketch-based input with large-scale vision-LLMs for grounding and generalization (Zheng et al., 20 May 2025, Zhang et al., 6 Jan 2026). Open limitations remain in scaling to highly dexterous or long-horizon tasks where static sketches underconstrain intent, or in settings demanding fine timing/force modulation.

A plausible implication is that future STRL systems will increasingly combine interactive sketch abstraction, stepwise marker-based feedback, and compositional subpolicy reuse, yielding agents that are not only data- and sample-efficient, but also robustly human-interpretable, transparent in their “thinking,” and seamlessly adaptable to new structural priors and interface modalities.