Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Junhao Shi, Zezheng Huai, Siyin Wang, Jia Chen, Yubang Wang, Zhaoye Fei, Hechang Chen, Jingjing Gong, Xipeng Qiu, Yu-Gang Jiang
• Venue: arXiv: cs.RO
• Published: 2026-06-25
• Collected: 2026-06-26
• URL: http://arxiv.org/abs/2606.27251v1
• PDF: https://arxiv.org/pdf/2606.27251v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers present a framework called OmniAct that aims to build persistent embodied agents for operation in unstructured environments. The framework integrates a multimodal semantic planner, an adaptive hierarchical memory, and an asynchronous visual preemption engine. These components work together to improve the end-to-end success of robotic tasks, maintain low resource consumption, and enhance the performance of mid-scale models.

What the Document Says

• The framework is called OmniAct
• OmniAct integrates a multimodal semantic planner, an adaptive hierarchical memory, and an asynchronous visual preemption engine
• These components work together to improve the success of robotic tasks
• OmniAct maintains low resource consumption
• OmniAct enhances the performance of mid-scale models

Key Concepts Mentioned

• Persistent embodied agents
• Unstructured environments
• Cyber (APIs, IoT)
• Physical (manipulation, navigation)
• Physical failures
• VLM-based planners
• VLA policies
• Hierarchical asynchronous architecture
• Planning
• Memory
• Verification
• Semantic loop
• End-to-end success
• Token consumption
• Accumulated interaction tokens
• Mid-scale models

Claims Made by the Document

• OmniAct improves the end-to-end success of robotic tasks
• OmniAct maintains low resource consumption
• OmniAct enhances the performance of mid-scale models

Evidence Used

• Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success
• OmniAct maintains near-flat token consumption over under 100k+ accumulated interaction tokens
• Elevates mid-scale open-weight models to proprietary-level performance

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

• The paper does not detail how OmniAct detects physical failures or recovers from them
• The term 'near-flat token consumption' is not clearly defined

Limitations Noted by the Agent

Agent interpretation — not from the source.

• It is unclear how the success of the tasks was measured
• The definition of 'consistent improvements' is not provided
• The specific robotic platforms and IoT devices used are not stated

Open Questions

• How does OmniAct detect physical failures and recover from them?
• What does 'near-flat token consumption' mean?
• What constitutes a 'long-horizon task'?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not state how OmniAct detects physical failures or recovers from them
• It's unclear what the authors mean by 'near-flat token consumption'
• The paper does not specify what constitutes a 'long-horizon task'

Raw Abstract / Excerpt

Building persistent embodied agents in unstructured environments demands unified orchestration of heterogeneous tools spanning both cyber (APIs, IoT) and physical (manipulation, navigation) domains, coupled with autonomous recovery from physical failures that inevitably arise over extended operation. Existing systems treat these as separate problems: VLM-based planners lack a unified cyber-physical action space, agent frameworks accumulate unbounded context that degrades temporal coherence, and VLA policies execute open-loop without detecting their own failures. We argue that persistent autonomy requires not a monolithic model but a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification. To this end, we present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces, an adaptive hierarchical memory with event-boundary-driven compression for sub-linear context growth, and an asynchronous visual preemption engine that closes the semantic loop during physical execution. Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success across all complexity levels, maintains near-flat token consumption over under 100k+ accumulated interaction tokens, and elevates mid-scale open-weight models to proprietary-level performance.