Source note2 min read
From Passive Video to Editable Experience: Physically Grounded Experience Synthesis for Embodied Intelligence
Source
arXiv
arxiv.org/abs/2607.26903v1
Source
- Source name: arXiv
- Source kind: paper
- Authors / organization: Jia Luo
- Venue: arXiv: cs.AI
- Published: 2026-07-29
- Collected: 2026-07-30
- URL: http://arxiv.org/abs/2607.26903v1
- PDF: https://arxiv.org/pdf/2607.26903v1
- DOI: UNKNOWN
Plain-Language Brief
Jia Luo introduces Pegasus, a framework to help robots learn from human manipulation videos by translating the human demonstrations into data that can be understood by robots. The method involves creating an intermediate representation graph and a robot planning graph, which allows for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, ensuring generalization beyond object identities. The framework also includes a closed-loop physics verifier to filter out invalid generations.
What the Document Says
- Pegasus is a low-resource framework for bridging the embodiment gap between human morphology and robot hardware.
- It constructs a graph-based intermediate representation from human videos, transforming it into a Robot Planning Graph for robot-conditioned video generation.
- The Task Graph is used, along with Affordance and Constraint Graphs.
- A hierarchical affordance latent space models the relationship between object states, affordances, and tasks.
- A closed-loop physics verifier filters invalid generations based on kinematic feasibility, collision constraints, and joint limits.
- The framework is evaluated across various egocentric manipulation benchmarks like GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments.
Key Concepts Mentioned
- Pegasus
- embodied AI
- human demonstrations
- graph-based intermediate representation
- Robot Planning Graph
- Task Graph
- Affordance and Constraint Graphs
- hierarchical affordance latent space
- kinematic feasibility
- collision constraints
- joint limits
Claims Made by the Document
- Pegasus can bridge the embodiment gap between humans and robots.
- It enables generalization beyond object identities.
Evidence Used
- The description of Pegasus' method and its components.
- Results from the evaluation across various benchmarks and robot embodiments.
Methods / Approach
UNKNOWN
Limitations Stated by the Authors
UNKNOWN
Limitations Noted by the Agent
Agent interpretation — not from the source.
UNKNOWN
Open Questions
UNKNOWN
Notes for Future Q&A
May include interpretation. Labeled where it does.
UNKNOWN
Raw Abstract / Excerpt
The key bottleneck in embodied AI is not model architecture but data. Although billions of human manipulation videos exist online, robots cannot directly learn from them due to the embodiment gap between human morphology and robot hardware. We introduce Pegasus, a low-resource framework that bridges this gap by translating human demonstrations into robot-learnable data through structured knowledge transfer. Instead of relying on raw video prompts, Pegasus constructs a graph-based intermediate representation: a Task Graph extracted from human videos is transformed through Affordance and Constraint Graphs into a Robot Planning Graph for robot-conditioned video generation. A hierarchical affordance latent space models the relationship between object states, affordances, and tasks, enabling generalization beyond object identities. A closed-loop physics verifier further filters invalid generations using kinematic feasibility, collision constraints, and joint limits. We evaluate Pegasus across a range of egocentric manipulation benchmarks, including GTEA Gaze+ and EPIC-KITCHENS-100, and diverse robot embodiments, assessing Task Correctness, Executability, State Consistency, and Learnability. Results demonstrate reliable cross-embodiment translation and show that robot data generation can be reframed from a hardware collection problem into a scalable, low-resource knowledge transfer problem.
