COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows

Abstract: Agents are increasingly used to construct workflows and assist humans in completing recurring tasks more efficiently. As these workflows become repeated and domain-specific, agent memory and reusable skills become increasingly important: agents should be able to recall workflow patterns, execution constraints, and user preferences from previous runs. We study this problem in workflow-based image generation and introduce COMFYCLAW, an agentic skill evolution harness for controlling ComfyUI workflows. COMFYCLAW formulates workflow construction as typed graph editing, exposes tools organized by construction stage, automatically reverts invalid edits, and uses a region-level vision-LLM (VLM) verifier to translate visual failures into actionable repair suggestions. The framework further evolves a progressively disclosed skill library, where trajectories, execution errors, and verifier feedback from previous runs are distilled into reusable Agent Skills. Across four benchmark splits, three agent models, and two image backbones, COMFYCLAW achieves the best average image-generation evaluation score across all six agent configurations, outperforming a verifier-only baseline without skill evolution. Human annotations further show that annotators prefer COMFYCLAW over variants without skill evolution. Our results suggest that skill evolution is an effective mechanism for improving agent reliability and performance in recurring visual workflow construction.

• The paper introduces SkillBridge, a framework that formalizes image generation as an MDP over graph edits, enabling robust recovery and efficient workflow evolution.
• It leverages verifier feedback and progressive skill mutations to optimize ComfyUI pipeline adjustments, achieving up to 10pt performance gains.
• The approach accumulates reusable procedural knowledge, with evolved skills representing benchmark-specific strategies in object count, spatial layout, and style control.

Self-Evolving Skill Harnesses for Graph-Based Image Generation

Introduction and Motivation

The shift from prompt-only interaction paradigms toward agent-based workflow construction in image generation introduces a need for agents to operate over explicit graph-structured interfaces, such as those exposed by ComfyUI. This domain uniquely demands agents capable of not only planning, editing, and repairing workflows in a sequential or iterative manner but also distilling persistent, reusable skills from prior tasks. "COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows" (2607.01709) proposes SkillBridge, an agentic framework for operating over ComfyUI workflow graphs, equipped with a closed-loop, verifier-driven skill evolution pipeline to facilitate both immediate error recovery and the long-term accumulation of actionable procedural knowledge.

Figure 1: The SkillBridge framework: agent-driven editing of ComfyUI workflow graphs, vision-LLM (VLM) feedback as actionable repair guidance, and the evolution of a reusable skill library.

Methodology

SkillBridge formalizes the workflow construction process as a Markov Decision Process (MDP) over graph edits, runtime, and verifier feedback, integrating three principal modules: skill-augmented graph construction, verifier-guided iterative refinement, and skill evolution with held-out validation.

Graph-Based Workflow Construction

Workflow construction proceeds as a sequence of typed edits—node/edge manipulations, parameter tuning—starting from an initial spine graph (minimal pipeline) and evolving toward a candidate solution. The agent retrieves relevant skills via a trigger-based router and applies them to efficiently navigate the high-dimensional space of possible workflow topologies. Skills are stored as SKILL.md artifacts with progressive disclosure to manage context window costs; only titles and descriptions are initially exposed, with full content retrieved on demand.

Verifier-Driven Iterative Refinement

A vision-LLM decomposes each input prompt into explicit binary requirements and calculates requirement-level scores and feedback (object presence/count, spatial relations, style, etc.), along with a holistic quality score. Structured, region-level critiques (failure descriptions and edit suggestions) inform the agent, which iterates on the workflow graph until satisfaction thresholds are met or a maximum number of modifications is reached.

Skill Evolution

SkillBridge’s distinguishing feature is its self-evolving skill harness. After batches of tasks, execution traces (actions, feedback, outcomes) are clustered into success and failure modes. For each cluster, candidate mutations (create, revise, reinforce, merge, delete) are proposed to the skill library. Each mutation is validated via a held-out cross-validation loop—new skills are admitted only if they improve or preserve performance on unseen prompts. Thus, skill aging, pruning, and specialization occur over time, and the agent continually accumulates procedural knowledge tailored to recurring workflow patterns.

Experimental Results

SkillBridge is evaluated across four challenging text-to-image benchmarks (GenEval2, DPG-Bench, OneIG-EN, OneIG-ZH) using three LLM agent models (Claude-Sonnet-4.5, Qwen-3.6-35B-A3B, Gemma-4-E4B-it) and two ComfyUI diffusion backbones. The agent operates externally, via unmodified ComfyUI APIs, and utilizes both predefined and evolved skills. Performance is principally measured with VLM-based metrics (Soft-TIFA geometric/arithmetic mean, VQAScore), with further qualitative assessments from human annotators.

Strong Numerical Results

SkillBridge achieves the highest average image-generation score across all agent-backbone combinations and all benchmarks, with substantial gains of up to 10 absolute points over no-refinement baselines and 4 points over non-evolving harness baselines. Notably, SkillBridge outperforms a verifier-only baseline without skill evolution in all tested configurations, indicating the impact of reusable skill acquisition.

Skill Utilization

Analysis shows that, on benchmarks requiring dense compositionality (e.g., DPG-Bench, GenEval2), up to 70% of skill invocations are evolved rather than predefined, demonstrating agent reliance on the self-evolved procedural library.

Figure 2: Iterative visual refinement under SkillBridge, with region-level VLM verifier feedback guiding structural graph edits and semantic alignment.

Qualitative analyses further reveal that only 39% of workflow edits are prompt text modifications; the remainder involve hyperparameter updates, graph topology changes (regional/mask structure, LoRA insertions), and other advanced interventions beyond the capabilities of vanilla prompt engineering.

Evolved Skill Patterns

The evolved skill library is benchmark-specific and not a collection of generic memories. For compositional benchmarks, skills encode strategies for object count preservation, spatial layout, and attribute binding. For stylized or domain-specific splits, the skills specialize in character arrangement, style declaration, and fine-grained material control. This specificity underpins high sample efficiency and quick recovery from known failure modes.

Implications and Future Directions

The results demonstrate that agentic workflow control with self-evolving skill harnesses induces robustness, sample efficiency, and transfer in graph-based creative domains. Agents not only adapt to unseen prompts but progressively encode domain best practices, execution constraints, and subtle recovery strategies in a persistent, validated form. The closed-loop between localized feedback and long-range skill accumulation represents a viable template for extending agentic control to ever more complex domains beyond image generation (e.g., video synthesis, multimodal reasoning, interactive GUI/task automation).

On the theoretical front, the integration of explicit skill libraries, progressive retrieval, and held-out cross-validation aligns with emerging directions in continual RL, lifelong learning, and procedural memory systems for autonomous agents. The MDP formulation with skill-conditional policies and dynamic action spaces is generalizable to other tool- or workflow-centric agent environments.

Current limitations are primarily in scalability to video workflows, where temporal consistency, scene planning, and exponential graph complexity demand even more advanced harnesses, skill discovery mechanisms, and context management.

Conclusion

SkillBridge establishes a robust, adaptive framework for workflow-based image generation, leveraging explicit agentic skills that evolve through continual, verifier-mediated experience. Through strong empirical results and detailed error analysis, the system demonstrates the value of integrating reusable skills, fine-grained feedback, and evolutionary validation into agentic control loops (2607.01709). This approach is well-positioned to inform future architectures for self-improving, lifelong agents operating over complex, structured environments.