Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

Source

arXiv

arxiv.org/abs/2607.05377v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Jiaqi Peng, Xiqian Yu, Delin Feng, Yuqiang Yang, Wenzhe Cai, Jing Xiong, Ganlin Yang, Jinliang Zheng, Jiafei Cao, Xueyuan Wei, Jiangmiao Pang, Yuan Shen, Tai Wang
  • Venue: arXiv: cs.RO
  • Published: 2026-07-06
  • Collected: 2026-07-07
  • URL: http://arxiv.org/abs/2607.05377v1
  • PDF: https://arxiv.org/pdf/2607.05377v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers introduce Cortex, an embodied agent framework designed to handle long-horizon manipulation tasks. The framework addresses the limitations of existing Vision-Language-Action (VLA) models by providing a customized planning interface that converts high-level VLM plans into low-level VLA execution. This is achieved by standardizing manipulation subtasks into 32 canonical skill primitives and enhancing the data generation pipeline with tractability principles. The framework was tested on various tasks, demonstrating improvements over monolithic baselines and even enabling zero-shot completion of unseen real-world long-horizon tasks.

What the Document Says

  • Cortex is a bidirectionally aligned embodied agent framework for long-horizon manipulation.
  • It addresses the Markovian nature issue in recent VLA models by providing a customized planning interface.
  • The planning interface conveys executable and tractable subtask plans from high-level VLM to low-level VLA.
  • 32 canonical skill primitives are used to standardize manipulation subtasks.
  • Tractability principles are injected into the data generation pipeline.
  • The framework uses over 4k hours of open-source video data and generates 30 hours of simulation data.
  • An event-balanced sampling strategy is devised for training data construction.
  • Cortex outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin.
  • Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks.

Key Concepts Mentioned

  • Vision-Language-Action (VLA) models
  • Long-horizon manipulation tasks
  • Bidirectionally aligned embodied agent framework
  • High-level VLM and low-level VLA
  • Canonical skill primitives
  • Tractability principles
  • Data generation pipeline
  • Training data construction
  • Event-balanced sampling strategy
  • Open-loop VLM and closed-loop system evaluations

Claims Made by the Document

  • Cortex addresses the limitations of existing VLA models.
  • Cortex outperforms monolithic baselines on Libero-long and RoboTwin.
  • Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks.

Evidence Used

  • The text mentions that Cortex addresses the Markovian nature issue in existing VLA models and provides a customized planning interface.
  • Results from Libero-long and RoboTwin evaluations are provided to show improvement over monolithic baselines.
  • The text states that Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex's efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.