Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

Source

Source name: arXiv
Source kind: paper
Authors / organization: Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
Venue: arXiv: cs.CV
Published: 2026-07-09
Collected: 2026-07-10
URL: http://arxiv.org/abs/2607.08497v1
PDF: https://arxiv.org/pdf/2607.08497v1
DOI: UNKNOWN

Plain-Language Brief

Researchers propose a new type of multimodal agent called the Cognitive-structured Multimodal Agent (CMA) that can understand, generate, and edit visual and textual information. The agent externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. It consists of a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller. To overcome the lack of turn-level retrieval supervision in existing datasets, they developed a Unified Scenario Engine that generates structured multi-turn conversations with fine-grained retrieval annotations for reinforcement learning. The researchers also present CMA-Harness, a tool-augmented deployment of the same cognitive structure, offering a more scalable and efficient paradigm for long-horizon multimodal agents.

What the Document Says

The paper proposes a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory.
The agent consists of a Perceptual Abstraction Engine, a Cognitive Retrieval Engine, and a Multimodal Executive Controller.
To address the lack of turn-level retrieval supervision in existing datasets, they developed a Unified Scenario Engine.
The researchers achieved 91.4% retrieval accuracy over 20-turn sessions with their 8B agent and surpassed 32B baselines by +8.2% while nearly halving per-turn inference time.
They also present CMA-Harness, a tool-augmented deployment of the same cognitive structure.

Key Concepts Mentioned

Cognitive-structured Multimodal Agent
Episodic Visual Memory
Perceptual Abstraction Engine
Cognitive Retrieval Engine
Multimodal Executive Controller
Unified Scenario Engine
CMA-Harness

Claims Made by the Document

The proposed agent can perform vision/language understanding and image generation/editing.
Their approach offers a more scalable and efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling.

Evidence Used

Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing.
Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s).
Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: https://github.com/caseclose/cma-harness ; Project page: https://caseclose.github.io/cma-harness/