# The More Accurate Future Prediction Becomes, the More Hesitant the Robot Gets

The more accurate future prediction becomes, the more hesitant the robot gets

Full article: https://haiguangboy.com/posts/omega-0

## Core Method

Future prediction is not treated as a video generation goal, but instead reconstructs an irrelevant compact prediction signal: given current observations and language instructions, predict compact future observation embeddings—no large video generator needed, nor test-time inversion of video back into actions. The key is having future-representing queries attend to action-representing queries in reverse, injecting predicted future visual dynamics into action representations—pushing the action branch to consider the future observations its own actions will cause.

Three-stage training: first fine-tune a VLM to express whole-body action semantics with discrete action tokens; then freeze the backbone and jointly train a video-action predictor, converting human demonstration data into executable robot action labels via simulation replay to resolve representation space incompatibility; finally, real-robot fine-tuning with a rolling-window consistency constraint during training.

## Key Results

A single multi-task policy (not task-specific ones) surpasses baselines like ACT, DP, and GR00T-N1.7 across 11 real-world household tasks, with the gap widening especially on articulated object interaction, bimanual coordination, and long-horizon navigation tasks. The multi-view version outperforms the pure first-person version; third-person observations are used only during training and do not affect deployment.

## Related Work Comparison

- Opposite direction · [LAWM: Why Action Labels Become a Burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Opposite direction · [No Imagination at Inference, Robots Still Hit 91.8%! Fast-WAM Debunks WAM's Core Assumption](https://haiguangboy.com/posts/fast-wam) `fast-wam_2026_07`: Training video objectives matters more than imagining the future at test time
- Opposite direction · [Tactile as Prediction Target Rather Than Observation Input](https://haiguangboy.com/posts/n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens) `n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens_2026_07`: Tactile as prediction target rather than observation input
- Opposite direction · [Tactile Prediction Should Only Be for Training, Not Shown to Actions](https://haiguangboy.com/posts/tacwam) `tacwam_anchor_guided_world_action_model_with_mechanics_aware_tactile_prediction_2026_08`: Coupling mechanism: motion queries attend to video queries in reverse, injecting predicted future visual dynamics into action representations contradicts AGT attention: action tokens are barred from reading future visual/tactile prediction tokens—an information isolation mechanism rather than action-conditioned dynamics
- Opposite direction · [Xu Huazhe Welcomes First Commercialization Partner, Former Meituan Drone Executive Joins](https://haiguangboy.com/posts/wx_top华人科创社_20260721) `wx_top华人科创社_20260721_2026_07`: Core design: future prediction is not a video generation goal but reconstructs an irrelevant compact prediction signal; action branch needs no pixel-level generation or test-time video-to-action conversion contradicts world model technical architecture: dual-model with video generator plus prediction nodes, including action injection/bootstrapped/persistent memory mechanisms
- Same direction · [An_Open_Foundation_Model_Towards](https://haiguangboy.com/posts/an_open_foundation_model_towards) `an_open_foundation_model_towards_2026_07`: Deployment: single forward pass ~0.14s (over 7Hz), predicting 25 action steps but executing only the first 8, with RTC-style hot start plus overlapping blending smoothing validates training-time RTC: masking the first d action tokens teaches the model smooth continuation

## Limitations

Only 10 trials per task, so a single success or failure swings results by 10 percentage points. Inference barely exceeds 7Hz, notably slower than comparable world models with latency optimizations. Cross-object, cross-scene, and human-data transfer generalization tests all have only qualitative descriptions, with no quantitative success rates reported.

## Author's Assessment (Not Paper Content, Cross-Paper Synthesis)

The most counterintuitive finding is an ablation: swapping in a different encoder for the current frame improves offline future prediction accuracy, yet real-robot execution degrades—the robot becomes overly static and hesitant. The reason is that encoder was designed for continuous multi-frame video, so single-frame input lacks sufficient information. The authors conclude that accurate future prediction alone does not guarantee good action generation, echoing findings from several other works: offline metrics cannot rank real deployment quality.

There is also a direct resonance. A route manifesto from a few days ago on whole-body intelligence pretraining paradigms argued: native humanoids should not be understood as robotic arms plus mobile bases, nor as a VLM with an action head appended; humanoid tasks have no clear boundary of first navigating then manipulating. This paper does not cite that manifesto, yet it is a fully running system that, from architecture design to problem framing, nearly empirically validates both claims—unified whole-body single-model joint generation of movement and manipulation is exactly what the manifesto called "native"; wiping tables and mopping floors requiring concurrent coordination is a concrete example of "no navigation-then-manipulation boundary." One is pure route judgment, the other grounds that judgment with data and ablations.