# What is closed is the kinematic loop, not the physical one

Data synthesized from a phone video outperforms real-robot collection: 62.5% vs 46.7%

Full article: https://haiguangboy.com/posts/video2robo

## Core method

↳ ① Skills are redefined as relative motion between objects, bypassing hand-to-gripper retargeting. The essence of pouring and sweeping lies in relative object displacement; action segmentation is determined by inter-object distance falling below 3 cm.
↳ ② 6D tracking relies entirely on off-the-shelf model pipelines: CoTracker3 tracks pixels, TRELLIS generates object 3DGS from a single view, then leverages the differentiability of 3DGS to optimize poses frame-by-frame. Generative 3DGS lacks absolute scale, compensated by VGGT's monocular depth.
↳ ③ Four augmentation types applied per frame: object rearrangement, tabletop background textures, lighting, and virtual camera perturbations.

## Key results

· Monocular 6D localization BOP AR mean 78.41, strongest baseline MegaPose 68.13.
· Generating a single demonstration averages 5.23 seconds (SkillGen 8.88, MimicGen 17.23), with 100% success across all six tasks; baseline failures stem from penetration and slipping.
· Real-robot Franka on six tasks: only varying object poses 62.50% vs real-robot collection 46.67%; adding texture, background, lighting, and viewpoint variations 58.33% vs 27.50%, with two simulation augmentation baselines dropping to just 5.83% and 2.50%.

## Related route comparisons

- Opposing route · [Liang Wenfeng's four-hour investor meeting transcript](https://haiguangboy.com/posts/liangwenfeng-world-model) `wx_elsewhere别处发生_20260722_2026_07`: Route bet: using kinematic consistency + 3DGS photorealism to bypass physics simulation contradicts ★ Divergence may stem from differing referents: Liang Wenfeng's parallel of "3D/video generation/world models" all fall under "renderers," while robotics requires "simulators"
- Opposing route · [Cross-embodiment transfer should happen at the experience layer](https://haiguangboy.com/posts/pegasus-editable-experience) `from_passive_video_to_editable_experience_physically_grounded_experience_synthes_2026_08`: Route bet: using kinematic consistency + 3DGS photorealism to bypass physics simulation contradicts Core claim: embodiment gaps should be bridged at the experience layer, not the pixel layer
- Opposing route · [Predictive features cannot be directly fed to diffusion models](https://haiguangboy.com/posts/leapbot-wa) `leapbot_wa_world_anchor_action_models_via_predictive_latent_alignments_2026_07`: Route bet: using kinematic consistency + 3DGS photorealism to bypass physics simulation contradicts Core claim: world modeling's utility for manipulation lies in abstract physical expectations, not photorealistic rendering
- Same route · [1 day, 1000 tasks, driven by inductive biases](https://haiguangboy.com/posts/mt3-thousand-tasks) `learning_a_thousand_tasks_in_a_day_2026_08`: Limited evaluation scope: six self-collected tasks, self-built simulation benchmark, real-robot baseline only 20 demonstrations validates Problem quantification: mainstream BC systems average 175–250 demonstrations per task, bimanual tasks require ~8K
- Same route · [LAWM: Why action labels become a burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Same route · [Switching logic should not be learned by the policy](https://haiguangboy.com/posts/dr-lfd) `decompose_and_reorganize_planning_with_primitives_and_visuomotor_policies_learne_2026_08`: Real-robot Franka: pure synthetic data 62.5% exceeds 46.67% from 20 real teleop demonstrations validates Results: simulation peg-in-hole ID 100% vs ACT 44%/DP 54%; on DexMimicGen, 100 demonstrations outperform baseline's 1000

## Boundaries

It avoids penetration not by computing more accurately, but by not performing collision solving. Coupled motion is a scripted kinematic process, lacking contact forces, friction, mass, and inertia. The paper states it only supports rigid bodies, with grasp poses still manually annotated; the calibration-free premise also requires roughly arranging the camera and tabletop into a synthetic layout, and validation is only on single-arm tabletop tasks across six scenarios.

## Author's assessment (not paper content, cross-paper synthesis)

This paper's route bet directly opposes several recent works. LeapBot-WA argues world modeling's utility lies in abstract physical expectations rather than photorealistic rendering; Genesis's official blog positions simulation as an evaluation engine rather than a data generator; "From Passive Video to Editable Experience" argues embodiment gaps should be bridged at the experience layer. Video2Robo does the opposite, and its real-robot numbers win.

It also doesn't overclaim: the paper writes "does not support deformable objects, because it requires learning dynamics models from video," effectively admitting rendering cannot replace physics. So this isn't a right-or-wrong debate—within the bandwidth of rigid tabletop objects, rendering plus kinematic scripts is indeed sufficient.

There's also a three-way divergence: how to use human videos. Video2Robo tracks relative object motion, CAIP tracks hand poses, and JoyAI-RA 0.5 learns a cross-embodiment latent action space.

Anchor for review in six months: whether it can escape rigid tabletop objects. If it scales to contact-dense assembly, deformables, and bimanual tasks, "no physics needed" holds; if not, the 62.5% only indicates the tasks are insensitive to physics.