GROW2: Grounding Which and Where for Robot Tool Use

Yuhong Deng1‡, Yuyao Liu2‡, David Hsu1
1National University of Singapore; 2Massachusetts Institute of Technology; ‡Equal Contribution
Conference on Robot Learning (CoRL), 2026
PDF

GROW2 is an open-world affordance-grounding method that enables robots to use everyday objects as improvised tools.

Abstract

Can the robot use a plate to cut a cake if no knife is available? Tool use greatly expands robot capabilities, but using everyday objects as improvised tools requires robots to address the challenge of open-world affordance grounding: selecting an open-category object to act as a tool and localizing its specific region of action. To this end, we introduce GROW2 (GROunding Which and Where), which leverages object parts as a natural abstraction to split the grounding process hierarchically into semantic and geometric levels, thus bypassing the need for data-heavy, end-to-end training. Semantically, GROW2 harnesses the commonsense reasoning of Vision-Language Models (VLMs) to parse a natural-language task instruction, select a suitable object as the tool, and identify task-relevant parts on the tool and the target object. Geometrically, vision foundation models then ground the selected parts into precise 3D regions from a single RGB-D image. Experiments on established benchmarks show that GROW2 outperforms state-of-the-art baselines on affordance prediction benchmarks. Further, it achieves zero-shot generalization over open-category objects and outperforms baselines in both simulated and real-world robot tool use experiments.

Video

Object parts as affordance indicators

Object parts, such as a blade, rim, and tip, provide a useful abstraction that captures both affordance semantics and the corresponding geometric structure. Using parts as an intermediate representation, we can decompose affordance grounding into semantic and geometric levels. This decomposition eliminates dependence on large-scale affordance annotations and lets GROW2 generalize to open-category objects.

Examples of object parts that indicate grasping and task-specific affordances.

GROW2

GROW2 overview. Given a task description and a single-view RGB-D observation, GROW2 first performs tool and part selection in semantic space to identify the tool and target objects along with their task-relevant parts. It then detects and grounds these parts into complete, function-faithful 3D affordance regions. Finally, the affordance regions serve as priors for motion planning to generate low-level actions for robot execution.

Overview of the GROW squared semantic selection, 3D grounding, and robot execution pipeline.


Results

Robot tool-use examples

Each example shows the robot manipulation video on top and the corresponding grounded 3D affordance regions below.

Pound

Manipulation

Affordance

Cut

Manipulation

Affordance

Lift

Manipulation

Affordance

Sweep

Manipulation

Affordance

Pour

Manipulation

Affordance

Open-world affordance grounding

GROW2 effectively grounds open-world affordances and generalizes to novel objects and broad tasks. Task-relevant affordance regions are highlighted in red.


Quantitative and qualitative results for open-world affordance grounding and robot tool use.

BibTeX

@article{deng2026grow2,
  title     = {{GROW$^2$}: Grounding Which and Where for Robot Tool Use},
  author    = {Yuhong Deng and Yuyao Liu and David Hsu},
  journal   = {arXiv preprint arXiv:2606.30632},
  year      = {2026},
}