Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

Abstract: Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/

• The paper presents a modular agent framework that decouples memory, retrieval, and executive control to address visual token explosion and cross-turn referencing issues in long-horizon dialogue.
• It leverages an Episodic Visual Memory system to compress and selectively retrieve visual information, achieving up to 91.4% retrieval accuracy in English and robust cross-turn consistency.
• Empirical results show the agent surpasses 32B parameter baselines with up to +18.4% gains on hard retrieval tasks and nearly halves inference time.

Cognitive-structured Multimodal Agent for Long-horizon Multimodal Understanding, Generation, and Editing

Motivation and Structural Limitations of Unified Multimodal Architectures

Unified multimodal models have demonstrated efficacy in vision-language understanding and generative tasks by autoregressively modeling interleaved text and visual tokens within a shared context window. However, these architectures exhibit pronounced limitations in long-horizon multimodal dialogue, particularly with visual token explosion and unreliable cross-turn referencing. Injecting all historical images into the context results in an exponential increase in visual tokens, constrains the reasoning window, and introduces semantic drift and retrieval errors in scenarios spanning multiple topics and interleaved task types.

Figure 1: A multi-turn multimodal dialogue produced by the agent, highlighting persistent memory, iterative editing, and accurate cross-turn visual retrieval.

Such limitations are evident even in strong unified models, where retrieval accuracy collapses on complex, long-range benchmarks. Parameter scaling fails to address context interference and inefficiencies in visual grounding, especially for iterative editing and understanding tasks requiring fine-grained recall.

Agent Architecture: Decoupling Memory, Retrieval, and Executive Control

The paper introduces a cognitive-structured agent that externalizes visual information into an Episodic Visual Memory (EVM), thereby allowing the agent to selectively reactivate relevant episodes during dialogue. The pipeline consists of three modular components:

• Perceptual Abstraction Engine (PAE): Compresses each incoming image into structured abstractions (tags, descriptive captions, thumbnails) for compact storage in EVM.
• Cognitive Retrieval Engine (CoRE): Performs cross-modal reasoning to retrieve only the visual episodes pertinent to the current turn.
• Multimodal Executive Controller (MEC): Integrates retrieved memory and dialogue context, inferring task intent and dispatching actions for understanding, generation, editing, composition, or chat.

  Figure 2: End-to-end pipeline of the proposed agent, showing visual abstraction, selective retrieval, and bounded token consumption regardless of dialogue length.

This modular decomposition ensures bounded per-turn visual token consumption and allows independent optimization of memory representation and retrieval policies. The architecture avoids repeatedly injecting historical images into the prompt, substantially reducing inference time and eliminating context overflow.

Scenario Engine, Benchmark Construction, and Reinforcement Optimization

To address the lack of turn-level retrieval supervision in existing datasets, the authors develop a Unified Scenario Engine capable of programmatically generating structured multi-turn conversations with explicit retrieval annotations. This engine simulates topic shifts, task interleaving, and hard-negative confounders, producing a rigorous benchmark (M2CA-Bench) stratified by retrieval difficulty and containing 2,000 annotated turns.

Figure 3: Closed-loop scenario engine for structured multi-turn conversation construction, combining programmatic user simulation, multimodal agent generation, retrieval verification, and difficulty stratification.

The training pipeline utilizes staged SFT and RL. RL directly rewards memory construction and retrieval policies for task-level retrieval success (rather than standalone captioning quality), using difficulty-weighted metrics for fine-grained optimization. PAE and CoRE modules are separately optimized, closing the loop and ensuring end-task utility.

Empirical Results: Retrieval, Generation, and Efficiency

The agent achieves 91.4% retrieval accuracy (English) and 89.6% (Chinese) over 20-turn sessions, surpassing parameter-scaled 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s $\rightarrow$ 12.7s). Performance improvements are accentuated at increased difficulty levels; the margin grows from +9.6% on Full to +18.4% on the Hard subset, demonstrating robustness in long-horizon recall.

Downstream Gemini-scored generation quality further reflects retrieval gains. The agent attains the highest scores (8.49 EN, 8.53 CN overall), with strongest improvements in understanding/chat and editing tasks. A 12% retrieval accuracy gain translates into a 0.60 (EN) and 1.43 (CN) quality improvement, indicating a strong propagation effect through the generative pipeline.

Qualitative analysis confirms accurate visual grounding, edit fidelity, and topic switching without semantic drift, outperforming all-context and text-memory baselines, which degrade sharply on long-range, hard-case scenarios.

Figure 4: Qualitative comparison illustrating multi-turn visual understanding, generation, and editing, with the cognitive agent adhering to user instructions and preserving cross-turn consistency.

Deployment: CMA-Harness and Tool Augmentation

The modular agent architecture is instantiated in CMA-Harness, a tool-augmented deployment integrating persistent multimodal memory, web search, image generation/editing/composition, and OpenAI-compatible APIs. The same PAE--EVM--CoRE--MEC structure governs task orchestration, memory management, and tool invocation, enabling practical open-ended multimodal workflows beyond benchmark-optimized dialogue.

Qualitative Examples Illustrating Long-horizon Interaction

Long-form dialog examples show the agent's ability to handle complex interactions—cross-topic visual comparison, iterative editing, composition, and recall of distant details—relying on persistent episodic memory for robust reasoning and generation.

Figure 5: Example Dialogue 1. Long-horizon multimodal interaction featuring generation, iterative editing, topic shifting, and question answering.

Figure 6: Example Dialogue 2. Environmental transformations and stable visual consistency across distant turns.

Figure 7: Example Dialogue 3. Complex object insertion, cross-scene composition, and detailed visual comparison.

Theoretical and Practical Implications

This work demonstrates that architectural specialization—explicit episodic memory, decoupled retrieval reasoning, and executive task planning—is crucial for scalable, efficient, and robust multimodal agents. The results directly contradict claims that parameter scaling and unified token-based autoregression suffice for long-horizon multimodal dialogue. The structured memory-centric paradigm enables persistent context, accurate retrieval, and efficient action planning, suggesting that future multimodal agent systems should adopt similar modularity.

The practical implications include deployability in open-ended environments (e.g., workflow assistants, interactive design tools, memory-augmented agents) and tractable scaling for industrial scenarios involving persistent memory, episodic recall, and tool augmentation.

Conclusion

The paper establishes that cognitive-structured multimodal agents, leveraging explicit episodic memory and modular retrieval and control, significantly outperform unified token-based architectures in long-horizon dialogue. The agent reaches state-of-the-art retrieval and generation metrics at lower inference cost, with persistent memory enabling robust editing, composition, and understanding across topics and tasks. These advances outline a scalable framework for future multimodal agent design, both for academic research and practical deployment (2607.08497).