Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

Source

Source name: arXiv
Source kind: paper
Authors / organization: Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
Venue: arXiv: cs.CV
Published: 2026-07-09
Collected: 2026-07-10
URL: http://arxiv.org/abs/2607.08497v1
PDF: https://arxiv.org/pdf/2607.08497v1
DOI: UNKNOWN

Plain-Language Brief

Researchers propose a new model called the Cognitive-structured Multimodal Agent. This agent is designed to understand and generate multimodal data (vision and language) in long-horizon dialogues more efficiently. It does this by storing visual information in an Episodic Visual Memory and reactivating relevant episodes during reasoning. The model consists of a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller. To train the model, the researchers developed a Unified Scenario Engine that generates structured multi-turn conversations with fine-grained retrieval annotations. The new model outperforms existing models in terms of retrieval accuracy and efficiency.

What the Document Says

The paper proposes a Cognitive-structured Multimodal Agent.
The agent externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning.
The agent consists of a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller.
To address the lack of turn-level retrieval supervision in existing datasets, they developed a Unified Scenario Engine.
They constructed a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall.
Their 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions and outperforms 32B baselines.

Key Concepts Mentioned

Cognitive-structured Multimodal Agent
Episodic Visual Memory
Perceptual Abstraction Engine
Cognitive Retrieval Engine
Multimodal Executive Controller
Unified Scenario Engine
long-horizon visual-dialogue benchmark

Claims Made by the Document

The proposed Cognitive-structured Multimodal Agent improves long-horizon multimodal dialogue.
The new model outperforms existing models in terms of retrieval accuracy and efficiency.

Evidence Used

Achieving 91.4% retrieval accuracy over 20-turn sessions and outperforming 32B baselines.

Methods / Approach

Externalizing visual information into an Episodic Visual Memory
Structured visual abstraction
Cross-turn memory retrieval
Autonomous task inference and action planning

Limitations Stated by the Authors

The paper does not discuss potential limitations of the Cognitive-structured Multimodal Agent.

Limitations Noted by the Agent

Agent interpretation — not from the source.

Potential limitations may include limited generalization to other tasks, reliance on specific data for training, and complexities in managing large amounts of memory.

Open Questions

How well does the Cognitive-structured Multimodal Agent perform on tasks beyond visual dialogue?
What are the potential challenges in managing large amounts of memory in the Episodic Visual Memory?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper focuses on the performance of the agent in long-horizon visual dialogues. For other tasks or potential limitations, further investigation may be required.

Raw Abstract / Excerpt

Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/