Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

Source

arXiv

arxiv.org/abs/2606.26981v1

Read the source

Source

Plain-Language Brief

Researchers propose a new method called In-Context Model Predictive Generation (ICMPG) for generating realistic human motions from textual descriptions. The method combines language-model planning with physical feedback to ensure the generated movements are both semantically accurate and physically plausible. ICMPG outperforms existing methods in terms of generality and realism.

What the Document Says

  • The paper presents a framework called In-Context Model Predictive Generation (ICMPG).
  • ICMPG integrates language-model planning with inference-time physical feedback for motion synthesis.
  • The framework reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules: Context-Aware Motion Generation (CAMG) and Model Predictive Generation (MPG).
  • CAMG uses an LLM to decompose textual commands and generate candidate motion sequences.
  • MPG evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence.
  • ICMPG's closed-loop refinement allows it to adapt motions to both input semantics and the simulated physical environment without task-specific policy retraining.
  • The experiments show that ICMPG generalizes robustly across diverse commands and produces more physically plausible and semantically faithful motions than baselines.
  • The framework can incorporate different LLM backbones, enabling versatile and controllable text-driven motion synthesis.

Key Concepts Mentioned

  • In-Context Model Predictive Generation (ICMPG)
  • Language Model (LLM)
  • Model Predictive Control (MPC)
  • Physical Simulation
  • Text-driven Motion Synthesis

Claims Made by the Document

  • ICMPG is a new method for generating realistic human motions.
  • ICMPG combines language-model planning with physical feedback.
  • ICMPG's closed-loop refinement allows it to adapt motions to both input semantics and the simulated physical environment.
  • ICMPG outperforms existing methods in terms of generality and realism.

Evidence Used

  • The paper provides a detailed description of the ICMPG framework.
  • The experiments across standard and zero-shot open-vocabulary settings are presented to support the claims.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.