Planner
Checks whether the visual scene contains enough information to solve the task. If not, it proposes candidate physical interactions.
PhysCaP : Grounding Code-as-Policy Agent with Physics-Informed Exploration
A physics-informed Code-as-Policy agent that actively measures hidden physical properties when passive vision is not enough.
1National Taiwan University
2NVIDIA Research
3Google DeepMind
4National Yang Ming Chiao Tung University
*Equal contribution
Abstract
We present PhysCaP , a physics-informed Code-as-Policy agent that enables active perception for robotic manipulation by explicitly exploring to infer latent physical properties that cannot be recovered from passive visual observation alone. Our key contributions are training-free physical property extraction modules that estimate object mass and stiffness from proprioception and a dual-agent exploration framework that selectively plans and prioritizes interactions based on expected information gain, achieving higher task success with fewer interactions and lower execution time than passive and naive interactive baselines across real-world and simulated benchmarks.
Method
PhysCaP augments Code-as-Policy agents with an exploration loop. The Planner detects missing physical information, the Prioritizer ranks candidate interactions by expected information gain, and executable code calls PhysX modules such as get_mass and get_stiffness.
Prioritizer
Filters implausible or redundant interactions and ranks the remaining candidates to reduce unnecessary robot actions.
Mass Measurement
Estimates object mass from joint-torque differences during a fixed lift trajectory, using the end-effector Jacobian.
Stiffness Measurement
Classifies stiffness from gripper displacement and normalized motor effort, using repeated measurements and majority vote.
PhysX Module Evaluation
PhysX (physical property extraction modules) estimate task-relevant latent properties from robot proprioception without additional sensing hardware.
Real-world and Simulated Tasks
Find Blue Cube
Reveal a hidden blue cube under one of several cups while avoiding infeasible cup lifts through visual and geometric reasoning.
Identify Empty Can
Use mass measurements to infer which can is empty when visual cues alone cannot reveal the hidden state.
Pick Ripe Avocado
Use stiffness measurements to identify a ripe avocado among visually similar candidates.
LIBERO Empty Can
Evaluate the same hidden-property reasoning pattern in a simulated manipulation environment.
Quantitative Results
9/10
Find Blue Cube SR
8/10
Identify Empty Can SR
9/10
Pick Ripe Avocado SR
| Method | Find Blue Cube | Identify Empty Can | Pick Ripe Avocado | ||||||
|---|---|---|---|---|---|---|---|---|---|
| SR | OI | Time | SR | OI | Time | SR | OI | Time | |
| CaP | 10/10 | 3 | 75.51 +/- 4 s | 2/10 | 2.5 | 71.71 +/- 20 s | 1/10 | 1 | 26.38 +/- 0 s |
| CaP+PhysX | 10/10 | 2.7 | 76.91 +/- 21 s | 7/10 | 3.9 | 268.08 +/- 16 s | 9/10 | 4 | 515.61 +/- 9 s |
| CaP+PhysX+Planner | 9/10 | 3 | 104.11 +/- 26 s | 7/10 | 4 | 274.14 +/- 77 s | 8/10 | 4.125 | 563.29 +/- 76 s |
| PhysCaP-joint | 8/10 | 2.125 | 65.91 +/- 23 s | 7/10 | 2.7 | 241.62 +/- 70 s | 7/10 | 2.71 | 384.39 +/- 111 s |
| PhysCaP | 9/10 | 1.33 | 40.48 +/- 15 s | 8/10 | 2.5 | 239.0 +/- 27 s | 9/10 | 2 | 300.47 +/- 53 s |
SR: success rate. OI: average number of exploratory object interactions on successful trials. Time: robot execution time.
Qualitative Results
Real World
Task Demonstrations
Real-world videos are clipped to highlight the task-relevant interaction sequence.
Task 1
Find Cube Under Cup
Task 2
Identify Empty Can
Task 3
Find Ripe Avocado
Simulation
LIBERO Task
PhysCaP
PhysCaP
Evaluate the complete simulation pipeline on hidden-property reasoning.
CaP+PhysX
CaP+PhysX
Run the simulated empty-can task with physical property extraction but no planner-guided exploration.
CaP+PhysX+Planner
CaP+PhysX+Planner
Use planner-guided physical exploration for the simulated empty-can task.
Baseline
OpenVLA Failure
Baseline
Pi-0.5 Failure
Baseline
MolmoAct-2 Failure
Baseline