Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
• Venue: arXiv: cs.CV
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08497v1
• PDF: https://arxiv.org/pdf/2607.08497v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have proposed a new type of multimodal agent called the Cognitive-structured Multimodal Agent. This agent is designed to improve long-horizon multimodal dialogue by externalizing visual information into an Episodic Visual Memory and selectively reactivating relevant episodes during reasoning. The agent consists of three components: Perceptual Abstraction Engine, Cognitive Retrieval Engine, and Multimodal Executive Controller. To address the lack of turn-level retrieval supervision in existing datasets, a Unified Scenario Engine was developed to generate structured multi-turn conversations with fine-grained retrieval annotations. The new agent surpasses 32B baselines by +8.2% while nearly halving per-turn inference time.

What the Document Says

• A new multimodal agent called Cognitive-structured Multimodal Agent has been proposed.
• The agent externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning.
• The agent consists of a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller.
• A Unified Scenario Engine was developed to address the lack of turn-level retrieval supervision in existing datasets.
• The new agent surpasses 32B baselines by +8.2% while nearly halving per-turn inference time.

Key Concepts Mentioned

• Cognitive-structured Multimodal Agent
• Episodic Visual Memory
• Perceptual Abstraction Engine
• Cognitive Retrieval Engine
• Multimodal Executive Controller
• Unified Scenario Engine
• turn-level retrieval supervision
• multimodal dialogue

Claims Made by the Document

• The proposed agent improves long-horizon multimodal dialogue.

Evidence Used

• The agent externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning.
• The new agent surpasses 32B baselines by +8.2% while nearly halving per-turn inference time.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/