# Predictive features cannot be fed directly into diffusion models

Predictive features cannot be fed directly into diffusion models

Full article: https://haiguangboy.com/posts/leapbot-wa

## Core method

↳ ① Insert an isotropic semantic autoencoder in the middle to compress JEPA features down to 96 dimensions. Beyond reconstruction and KL, the key is slice-wise isotropy regularization—the authors explicitly note that in highly structured semantic spaces, KL alone cannot prevent dimensional collapse.
↳ ② The world model branch only observes intent, not actions: it conditions on language instructions and proprioception, without step-by-step low-level motor commands, preventing semantic representations from collapsing into a control shortcut.
↳ ③ Asymmetric masking prevents the world model branch from seeing action tokens, while the action branch can read semantic tokens. During training, the former acts as a privileged expert; at deployment, the entire branch is pruned, leaving only a precomputed semantic cache.

## Key results

· The ablation of that regularization is most telling: latent rank 38.1→92.3, gradient variance 1.25→0.18; success rate rises from 42.5% with the base encoder, 58.2% with the unregularized adapter, to 71.3%
· RoboTwin 2.0 average 91.46%, randomized 92.34%; LIBERO 97.3%; LIBERO-Plus zero-shot 73.1%—and without large-scale robot trajectory pretraining

## Related route comparison

- Opposite route · [It's not "whether to have a world model," it's "whether what it outputs has structure"](https://haiguangboy.com/posts/robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode) `robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode_2026_07`: It's not "whether to have a world model," it's "whether what it outputs has structure"
- Opposite route · [No need to imagine the future at inference time—robots still reach 91.8%! Fast-WAM dismantles WAM's core assumption](https://haiguangboy.com/posts/fast-wam) `fast-wam_2026_07`: Training video objectives matters more than imagining the future at test time
- Opposite route · [Xu Huazhe welcomes first commercialization partner, former Meituan drone executive joins](https://haiguangboy.com/posts/wx_top华人科创社_20260721) `wx_top华人科创社_20260721_2026_07`: Key design: world model only sees intent, not actions—deliberately stripping low-level motor commands to prevent collapse into control shortcuts contradicts world model technical architecture: video generation model + prediction node dual model, with action injection/bootstrapped/persistent memory three mechanisms
- Same route · [LAWM: Why action labels become a burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Same route · [T-Rex: Why touch needs separate modeling](https://haiguangboy.com/posts/t_rex_tactile_reactive_dexterous_manipulation) `t_rex_tactile_reactive_dexterous_manipulation_2026_07`: T-Rex: Why touch needs separate modeling
- Same route · [π0.5 shudders in place when failing to grab a spoon, but Orca goes further with physics intuition learned from watching videos](https://haiguangboy.com/posts/orca) `orca_2026_07`: The key to a world model is a readable state

## Boundaries

"Zero overhead" refers only to inference. Training requires 24 H200s, two 30-layer DiTs running simultaneously, and three-stage progressive training to prevent collapse. The authors note in limitations that performance is capped by the representation quality of the underlying predictive foundation model. Real-robot tests only cover a single UR5 picking and placing fruits and vegetables; the main text gives no success rate and no baseline comparison.

## Author's judgment (not from the paper, cross-paper synthesis)

"Auxiliary signals cannot be plugged into the backbone without constraints; they need a structurally restricted position"—this idea has recurred recently across very different directions: some implement it as residual terms, some as prediction targets, some add sparse routing to dedicated pathways, and this paper uses attention masks to control who sees what. Touch, sensor calibration, world models—the problem domains differ widely, yet the solutions converge to the same shape.

What remains undecided are two other points. First, whether to go through pixels: another study found that "rendering numerical values as images is more effective than injecting coordinates via MLP," citing the ability to leverage pretrained visual priors; this paper argues for leaving pixels entirely. Both sides have data, and the directions are opposite.

Second is when imagination should occur. This paper prunes the world model branch at deployment, while navigation-focused work advocates "think before acting." However, that line of work itself admits gains mainly appear in scenarios with high occlusion and dynamic density. So pruning for manipulation is reasonable, retaining for navigation is reasonable—the divergence may not be about architectural correctness, but about whether task uncertainty is high enough.