VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrazza, Guanya Shi, Karen Liu
• Venue: arXiv: cs.RO
• Published: 2026-06-29
• Collected: 2026-06-30
• URL: http://arxiv.org/abs/2606.30645v1
• PDF: https://arxiv.org/pdf/2606.30645v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers present a method to learn humanoid locomotion and manipulation tasks by synthetically generating vision-language-kinematics (VLK) supervision in reconstructed scenes. They produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. The policy is then converted into actions on the physical humanoid robot Unitree G1.

What the Document Says

• The research addresses the issue of perception-based humanoid loco-manipulation, where connecting egocentric observations and task instructions to whole-body motion is required. However, existing data sources lack complete tuples (egocentric images, language commands, and robot-compatible kinematic trajectories) at scale.
• The authors propose a solution by synthesizing VLK supervision in reconstructed scenes using 3D Gaussian Splatting to create metric-scale indoor environments, generating navigation and object-interaction trajectories, and rendering paired egocentric observations afterwards.
• They produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. The policy is then converted into actions on the physical humanoid robot Unitree G1.
• The researchers evaluate their method on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation.

Key Concepts Mentioned

• perception-based humanoid locomotion
• humanoid manipulation
• egocentric observations
• task instructions
• whole-body motion
• synthetic interactions
• reconstructed scenes
• 3D Gaussian Splatting
• vision-language-kinematics (VLK) supervision
• Unitree G1

Claims Made by the Document

• Existing data sources lack complete tuples for perception-based humanoid loco-manipulation at scale.
• Synthesizing VLK supervision in reconstructed scenes is an effective solution.
• The synthesized interactions provide effective supervision for sim-to-real perception-based humanoid loco-manipulation.

Evidence Used

• No existing data source provides this complete tuple at scale
• They produce 48,000 paired trajectories with no human intervention and train a VLK policy
• The researchers evaluate their method on the physical Unitree G1 performing navigation and single-object transport

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation. Project Website: https://vision-language-kinematics.github.io/