Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

GROW$^2$: Grounding Which and Where for Robot Tool Use

Source

arXiv

arxiv.org/abs/2606.30632v1

Read the source

Source

Plain-Language Brief

Researchers introduce GROW$^2$, a system that helps robots use tools creatively by grounding which object to use as a tool and where to apply it, even when the object is not intended for that purpose. This system uses commonsense reasoning and vision foundation models to parse natural-language instructions and identify task-relevant parts of objects, then ground these parts into precise 3D regions from an RGB-D image. GROW$^2$ outperforms current state-of-the-art baselines on affordance prediction benchmarks and demonstrates zero-shot generalization over open-category objects in both simulated and real-world experiments.

What the Document Says

  • The paper introduces a system called GROW$^2$ for robot tool use.
  • GROW$^2$ uses object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels.
  • Semantically, GROW$^2$ uses commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction.
  • Geometrically, vision foundation models in GROW$^2$ ground the selected parts into precise 3D regions from a single RGB-D image.
  • The researchers claim that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks.
  • GROW$^2$ achieves zero-shot generalization over open-category objects.
  • The system demonstrates its effectiveness in both simulated and real-world robot tool use experiments.

Key Concepts Mentioned

  • robot tool use
  • open-world affordance grounding
  • object parts
  • Vision-Language Models (VLMs)
  • RGB-D image
  • affordance prediction benchmarks
  • zero-shot generalization

Claims Made by the Document

  • GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks.
  • GROW$^2$ achieves zero-shot generalization over open-category objects.

Evidence Used

  • Experiments on established benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks.
  • The system demonstrates its effectiveness in both simulated and real-world robot tool use experiments.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but to use tools creatively beyond their intended functions, the robot faces the challenge of $\textit{open-world affordance grounding}$: select an open-category object to act as a tool and localize its specific region of action. To this end, we introduce GROW$^2$ (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW$^2$ harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW$^2$ outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.