# MoCaP: Zero-Shot Mobile Manipulation Driven by Human Videos

MoCaP: Zero-Shot Mobile Manipulation Driven by Human Videos

Full article: https://haiguangboy.com/posts/mocap

## Core Method

MoCaP organizes everyday videos into reusable scene memory and approach priors, then a closed-loop code agent combines real-time observations to invoke navigation, planning, and grasping tools. All learned components are frozen with no task-specific training; the focus is on connecting human experience to the robot's execution constraints.

↳ ① Extracting "where to look" from everyday recordings
Before receiving a downstream instruction, the system first organizes locations, objects, connectivity relations, and reference frames from the videos into scene memory that can be reused across tasks. When a task starts, the advisor combines the memory, the instruction, and the robot's initial view to propose likely locations and navigation subgoals.

↳ ② Extracting "which side to approach from" from human interactions
The recordings also preserve preparatory cues from before a person reaches out. The robot borrows the approach direction, aligns geometric scale using real-time depth, and then selects a docking distance based on its own size and kinematics, searching for nearby base positions and orientations. Human experience provides the direction; the specific stance is determined by the robot's constraints.

↳ ③ Connecting experience to actions with closed-loop code
The code agent generates short Python programs based on observations, invoking Nav2 and grasping primitives. After the base adjusts, it re-perceives and then uses cuRobo to plan full-arm motion; if a call fails, subsequent programs are interrupted, diagnostics are returned to the agent, and the next step is revised. Navigation, stance, and grasping are chained together by feedback.

## Key Results

Complete grasping of the cardboard box: 16/20; can: 8/10; both at 80%. Navigation to the blue trash bin: 10/10; the variant without video priors: 7/10. A grasp success requires picking up the specified object and holding it stably.
Both grasping baselines failed, but the grasping and planning backends also differ, so the entire gap cannot be attributed to video experience.

## Comparison with Related Lines

- Opposite line · [An_Open_Foundation_Model_Towards](https://haiguangboy.com/posts/an_open_foundation_model_towards) `an_open_foundation_model_towards_2026_07`: All learned components frozen, no task-specific training contradicts joint co-training on heterogeneous human-robot data being a structurally suboptimal approach
- Opposite line · [The more accurate the future prediction, the more sluggish the robot becomes](https://haiguangboy.com/posts/omega-0) `omega_0_a_latent_predictive_world_action_model_for_concurrent_humanoid_loco_mani_2026_08`: Re-observing after base movement, then planning full-arm motion contradicts Problem framing: existing VLAs are either arm-centric or treat the base and arm as two uncoordinated subsystems
- Same line · [1,000 tasks in 1 day, thanks to inductive bias](https://haiguangboy.com/posts/mt3-thousand-tasks) `learning_a_thousand_tasks_in_a_day_2026_08`: Zero-shot still relies on calibration, depth, and off-the-shelf execution modules validates Perception dependency: vision only, single camera, no touch, relies on accurate segmentation
- Same line · [DexTouch-WM: Human tactile pretraining](https://haiguangboy.com/posts/dextouch-wm) `dextouch_wm_learning_action_conditioned_tactile_world_models_from_human_touch_fo_2026_09`: Zero-shot still relies on calibration, depth, and off-the-shelf execution modules validates Human data is not a direct transfer without hardware or robot data
- Same line · [Introducing Gemini Robotics ER 2](https://haiguangboy.com/posts/google_blog_gemini_robotics_er_2_20260730) `google_blog_gemini_robotics_er_2_20260730_2026_07`: The code agent uses execution feedback to coordinate navigation and grasping validates Direct implication for navigation practitioners: navigation is not absorbed by the base; it is demoted to a tool invoked by the base
- Same line · [MINT_A_Unified_Model_for_World_Space_Camera_and_Hand_Motion_Estimation_from_Scal](https://haiguangboy.com/posts/mint_a_unified_model_for_world_space_camera_and_hand_motion_estimation_from_scal) `mint_a_unified_model_for_world_space_camera_and_hand_motion_estimation_from_scal_2026_09`: VGGT relative poses obtain scale through robot depth validates The judgment behind two-stage training: learning a stable hand-camera coupled geometry and correcting the absolute scale of camera trajectories are two subproblems that can be solved separately—the former needs large amounts of diverse but noisy data, the latter needs only a small amount of high-precision data

## Boundaries

⚠️ Costs and boundaries
The evaluation covers only one environment, one recording, and three targets, and the manipulation is mainly grasping and lifting. Zero-shot still requires calibration, real-time depth, kinematics, and off-the-shelf models. Some trial-count statements in the original text conflict, and readiness checks cannot be treated as actual grasp success rates.

## Author's Assessment (not part of the paper; cross-paper synthesis)

💡 Engineering takeaway
The use of human videos can extend to memory and approach experience invoked directly at execution time. This echoes the discussion of executor startup conditions in "Towards Agentic Robots": choosing the right skill is not enough; the robot must also be brought into a state where the skill can be executed.
MoCaP demonstrates a path for reusing experience without updating model weights. Whether it can scale still depends on whether the full manipulation success rate benefits stably after changing environments, changing recordings, and unifying the backend.