Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics

Source

arXiv

arxiv.org/abs/2606.26981v1

Read the source

Source

Plain-Language Brief

The paper proposes a new method called In-Context Model Predictive Generation (ICMPG) for generating human motions based on textual descriptions in digital applications. The method combines language model planning with physical feedback to ensure both semantic fidelity and physical realism, addressing the trade-off issue present in existing methods.

What the Document Says

  • The paper presents a new framework called In-Context Model Predictive Generation (ICMPG).
  • ICMPG is designed for synthesizing human motion from textual descriptions for immersive digital applications.
  • Existing methods face a trade-off between semantic fidelity and physical realism in synthesizing human motion.
  • Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions but often generate motions that violate physical constraints.
  • Physics-aware models improve realism through simulation or control but struggle with semantic complexity, fine-grained instructions, and novel concepts.
  • ICMPG integrates language-model planning with inference-time physical feedback to address the gap between LLM-based approaches and physics-aware models.
  • The framework is divided into two modules: Context-Aware Motion Generation (CAMG) and Model Predictive Generation (MPG).
  • CAMG uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens.
  • MPG evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps.
  • ICMPG's closed-loop refinement allows it to adapt motions to both input semantics and the simulated physical environment without task-specific policy retraining.
  • The experiments showed that ICMPG generalizes robustly across diverse commands and produces motions more physically plausible and semantically faithful than baselines on benchmarks.

Key Concepts Mentioned

  • human motion synthesis
  • large language models
  • physics-aware models
  • Model Predictive Control (MPC)
  • semantic fidelity
  • physical realism

Claims Made by the Document

  • ICMPG generalizes robustly across diverse commands.
  • ICMPG produces motions that are more physically plausible and semantically faithful than representative baselines.

Evidence Used

  • Results from extensive experiments across standard and zero-shot open-vocabulary settings.

Methods / Approach

  • In-Context Model Predictive Generation (ICMPG) framework
  • Context-Aware Motion Generation (CAMG) module
  • Model Predictive Generation (MPG) module

Limitations Stated by the Authors

  • The paper does not discuss the potential challenges in scaling ICMPG to handle large numbers of textual commands or complex physical environments.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The effectiveness of ICMPG may be influenced by the quality and diversity of the training data used for the LLM.
  • The performance of ICMPG may degrade in situations where the textual descriptions are ambiguous, incomplete, or contain errors.

Open Questions

  • How can ICMPG be improved to handle large numbers of textual commands or complex physical environments?
  • What measures can be taken to ensure that ICMPG performs well with ambiguous, incomplete, or error-containing textual descriptions?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The authors focus on improving the trade-off between semantic fidelity and physical realism in synthesizing human motion. They achieve this by integrating language model planning with physical feedback.
  • The authors do not discuss how they ensure the diversity of training data for their LLM.

Raw Abstract / Excerpt

Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and physical realism. Large language model (LLM)-based approaches can interpret diverse open-vocabulary instructions and compose high-level action plans, but they often generate motions that violate physical constraints. Physics-aware models improve realism through simulation or control, but they struggle with semantic complexity, fine-grained instructions, and novel concepts. To address this gap, we propose In-Context Model Predictive Generation (ICMPG), a framework that integrates language-model planning with inference-time physical feedback. ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules. The Context-Aware Motion Generation (CAMG) module uses an LLM as a planner to decompose textual commands and generate candidate motion sequences from motion tokens. The Model Predictive Generation (MPG) module evaluates these candidates through physical simulation and semantic alignment, estimates a composite reward, and selects the best sequence to guide subsequent generation steps. Unlike open-loop generation, this closed-loop refinement enables ICMPG to adapt motions to both the input semantics and the simulated physical environment without task-specific policy retraining. Extensive experiments across standard and zero-shot open-vocabulary settings show that ICMPG generalizes robustly to diverse commands and produces motions that are more physically plausible and semantically faithful than representative baselines on the evaluated benchmarks. The framework bridges semantic interpretation and physical simulation while remaining flexible enough to incorporate different LLM backbones, enabling more versatile and controllable text-driven motion synthesis.