# World models misbehave because latent actions are contaminated

World models misbehave because latent actions are contaminated

Full article: https://haiguangboy.com/posts/cd-lam

## Core method

↳ ① Three debiasing objectives: SAM3 foreground masks for ego-centric weighted reconstruction; caption verbs clustered into 12 coarse primitives for pairwise contrast, keeping same-primitive transitions close across scenes; free-bits KL with zero-transition calibration, compressing latent magnitudes for repeated-frame inputs to a small fraction of normal transitions.
↳ ② Three stages: 1k steps to debias LAM, 2k steps to adapt the world model to the new latent space, then a lightweight MLP bridge to executable actions. The world model's architecture, latent dimensions, and conditioning format remain unchanged throughout.

## Key results

· Encoder audit: zero-transition response median 0.527→0.043, shortcut leakage 0.151→0.014, while cross-segment same-primitive pair similarity barely moves—targeted debiasing, not blanket shrinkage
· Action following error: latent action stage down 42% at 2B, 26% at 14B; after bridging to robot actions, down another 35%, 30%
· Residual motion under zero-action intervention: 14B drops from 9.36 to 2.18
· Adaptation updates reduced by over 12x; 1 hour of debiasing data captures about 80% of the benefit from the 1000-hour tier

## Related work comparison

- Same line · [An_Open_Foundation_Model_Towards](https://haiguangboy.com/posts/an_open_foundation_model_towards) `an_open_foundation_model_towards_2026_07`: Efficiency and data stratification: over 12x fewer adaptation updates; 1 hour of debiasing data achieves 80% of the benefit, validating Ψ0's extreme data efficiency: 800h+30h surpasses a 10x data baseline by 40%+
- Same line · [LAWM: Why action labels become a burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Same line · [It's not "whether to have a world model," it's "whether what it outputs has structure"](https://haiguangboy.com/posts/robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode) `robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode_2026_07`: It's not "whether to have a world model," it's "whether what it outputs has structure"
- Same line · [T-Rex: Why touch needs separate modeling](https://haiguangboy.com/posts/t_rex_tactile_reactive_dexterous_manipulation) `t_rex_tactile_reactive_dexterous_manipulation_2026_07`: T-Rex: Why touch needs separate modeling
- Same line · [π0.5 fails to grab a spoon and trembles in place, but Orca goes further with physics intuition learned from watching videos](https://haiguangboy.com/posts/orca) `orca_2026_07`: The key to a world model is readable states
- Same line · [Transcript of Liang Wenfeng's four-hour investor meeting](https://haiguangboy.com/posts/liangwenfeng-world-model) `wx_elsewhere别处发生_20260722_2026_07`: Three-stage pipeline: first debias LAM (1k steps) → then debias ACWM (2k steps) → finally connect executable action bridge, validating ★★core judgment: the implicit premise of "world models are irrelevant to intelligence ceilings" is that "the data pipeline is clear"—for LLMs it's clear so it's a detour, for robots it's not so it's a bridge

## Limitations

The debiasing stage additionally uses SAM3 masks and primitive labels, which the baseline lacks, so "only changing the latent action space" doesn't hold at the supervision-signal level, and no controlled comparison giving the baseline the same supervision was done. Core metrics are author-defined, with multiple evaluation setups independent and explicitly not cross-comparable across tables. All conclusions come from offline video metrics, without closed-loop integration into planning or policy evaluation. Training used 96 H100s.

## Author's take (not from the paper, cross-paper synthesis)

Looking at the three entries side by side, they're three layers of the same thing. The outermost is an existing judgment: whether world models are useful is the wrong question—conditional signals and output structure determine whether they help or hinder. The middle layer is another paper's empirical finding: feeding unstructured latent variables to a VLA hurts performance, while swapping in structured intermediate representations steadily improves it. This paper fills in the innermost layer—why performance drops, and how to fix it.

What's more noteworthy is the counterintuitive finding: from 2B to 14B, pixel metrics improve while action following worsens. Scaling parameters amplifies visual capability, not controllability. This parallels "stricter segmentation beats looser segmentation"—alignment quality determines whether data becomes capability.

There's also a cross-domain easter egg: the camera-conditioning paper found that backgrounds leak camera pose, making viewpoint invariance look real when it's actually a shortcut. The same mechanism bites twice in unrelated places—anytime a network is left to infer some quantity from pixels on its own, it's worth checking whether it's just copying the background.