Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Junhao Shi, Zezheng Huai, Siyin Wang, Jia Chen, Yubang Wang, Zhaoye Fei, Hechang Chen, Jingjing Gong, Xipeng Qiu, Yu-Gang Jiang
• Venue: arXiv: cs.RO
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27251v1
• PDF: https://arxiv.org/pdf/2606.27251v1
• DOI: UNKNOWN

Plain-Language Brief

The paper presents OmniAct, a framework designed to create persistent embodied agents that can operate autonomously in unstructured environments. It addresses the challenges of integrating cyber (APIs, IoT) and physical (manipulation, navigation) tools, recovering from physical failures, and maintaining temporal coherence. The framework includes a multimodal semantic planner, an adaptive hierarchical memory, and an asynchronous visual preemption engine to enhance the agent's performance in real-world long-horizon tasks.

What the Document Says

• The paper discusses the challenge of creating persistent embodied agents that can operate in unstructured environments.
• It argues for a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification for achieving persistent autonomy.
• OmniAct is presented as a framework that integrates a multimodal semantic planner, an adaptive hierarchical memory, and an asynchronous visual preemption engine.
• The framework aims to improve the agent's performance in real-world long-horizon tasks on two robotic platforms coordinating four IoT devices.
• Across 40 such tasks, OmniAct consistently improves end-to-end success and maintains near-flat token consumption over under 100k+ accumulated interaction tokens.

Key Concepts Mentioned

• persistent embodied agents
• unified action spaces
• hierarchical asynchronous architecture
• multimodal semantic planner
• adaptive hierarchical memory
• asynchronous visual preemption engine
• real-world long-horizon tasks

Claims Made by the Document

• Persistent autonomy requires a hierarchical asynchronous architecture.
• OmniAct can achieve consistent improvements in end-to-end success and maintain near-flat token consumption.

Evidence Used

• Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success...

Methods / Approach

• OmniAct framework integrates multimodal semantic planner, adaptive hierarchical memory, and asynchronous visual preemption engine.

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not provide information about potential limitations of the presented framework.

Open Questions

• What are the specific benefits of using OmniAct compared to existing systems?
• How does the asynchronous visual preemption engine work in practice?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• It is unclear what 'VLM-based planners' and 'VLA policies' refer to.
• The term 'token consumption' might need further explanation.

Raw Abstract / Excerpt

Building persistent embodied agents in unstructured environments demands unified orchestration of heterogeneous tools spanning both cyber (APIs, IoT) and physical (manipulation, navigation) domains, coupled with autonomous recovery from physical failures that inevitably arise over extended operation. Existing systems treat these as separate problems: VLM-based planners lack a unified cyber-physical action space, agent frameworks accumulate unbounded context that degrades temporal coherence, and VLA policies execute open-loop without detecting their own failures. We argue that persistent autonomy requires not a monolithic model but a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification. To this end, we present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces, an adaptive hierarchical memory with event-boundary-driven compression for sub-linear context growth, and an asynchronous visual preemption engine that closes the semantic loop during physical execution. Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success across all complexity levels, maintains near-flat token consumption over under 100k+ accumulated interaction tokens, and elevates mid-scale open-weight models to proprietary-level performance.