In-Context Robot Learning with VLM Agents

Source facts

The paper introduces GPT-Policy, a framework combining a context compiler, a vision-language model that proposes robot-tool actions, and a constrained controller that verifies and executes actions. The authors report real-robot trials in which human video demonstrations improve task completion without gradient updates, with aligned action references providing additional gains on contact-sensitive tasks.

Analysis

The work treats context as a deploy-time adaptation substrate for robots: demonstrations and interaction feedback are compiled into action-relevant transitions, then checked by a controller before execution. This is a systems pattern for grounding general-purpose VLMs in physical behavior rather than a claim that open-ended robot learning is solved.

Uncertainty/Doubt

• This is an arXiv preprint; the reported real-robot results and comparative claims were not independently reproduced here.