# World Models: Should We Imagine at Inference Time?

SLIM, JEPA-WAM all say don't do the work at inference—this ablation shows: remove that step, and the refiner performs worse than no refinement at all

Full article: https://haiguangboy.com/posts/omega-eva

## Core Method

Have the policy first propose an action, feed it back into a frozen world model to get imagined consequences, then train a refiner using the current state, imagined consequences, and original proposal to rewrite the action. Only one latent-space forward pass, no image decoding, no multi-step rollout.
Three stages: Stage 1 learns an action-conditioned world model, also producing a current representation decoupled from actions—feeding expert, zero, random, and shuffled actions yields numerically identical outputs (difference 0.000, cosine similarity 1.000). Stage 2 generates initial proposals from this representation. Stage 3 freezes the first two stages, training the refiner only on the policy's own proposals, with training and inference sharing the same data flow.

## Key Results

0.8B, zero robot pretraining, LIBERO 97.9%, refined to 98.6%, surpassing multiple larger and pretrained models.
The component ablation is most telling: removing the imagined-future branch drops to 97.2%—lower than the no-refinement baseline of 97.9%. Removing the proposal branch (can see the future but doesn't know which action caused it) drops to 96.0%. Neither branch is decorative.
1-step denoising matches 5-step (98.8 vs 98.4), cutting latency from 45ms to 25ms.

## Related Approaches Comparison

- Opposite route · [JEPA-WAM: Innovative method makes world models better at understanding state changes](https://haiguangboy.com/posts/jepa-wam) `jepa_wam_learning_vision_language_action_policies_with_joint_embedding_world_modeling_2026_08`: Core claim: world models shouldn't assist during training or act as external simulators, but should serve as verifiers inside action generation contradicts Core claim: should predict transition relations, not absolute future states
- Opposite route · [LAWM: Why action labels become a burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Opposite route · [No need to imagine the future at inference, robots still hit 91.8%! Fast-WAM dismantles WAM's core assumption](https://haiguangboy.com/posts/fast-wam) `fast-wam_2026_07`: Training video objectives matters more than imagining the future at test time
- Opposite route · [Tactile prediction should only be used for training, not shown to actions](https://haiguangboy.com/posts/tacwam) `tacwam_anchor_guided_world_action_model_with_mechanics_aware_tactile_prediction_2026_08`: Core claim: world models shouldn't assist during training or act as external simulators, but should serve as verifiers inside action generation contradicts Core principle: future tactile signals only serve as training supervision, never enter deployment-time action inputs
- Opposite route · [Predictive features can't be fed directly to diffusion models](https://haiguangboy.com/posts/leapbot-wa) `leapbot_wa_world_anchor_action_models_via_predictive_latent_alignments_2026_07`: Problem breakdown: existing world model uses fall into three categories, none let candidate actions be tested and corrected by their own imagined results contradicts Key design: world model only sees intent, not actions—deliberately depriving low-level motor commands to prevent collapsing into a control shortcut
- Opposite route · [Lower prediction error, but representation already collapsed](https://haiguangboy.com/posts/slim-05b) `slim_05b_learning_action_grounded_predictive_latents_for_robot_manipulation_2026_08`: Core claim: world models shouldn't assist during training or act as external simulators, but should serve as verifiers inside action generation contradicts Mechanism claim: one-way prediction isn't enough; both directions must jointly constrain the representation

## Boundaries

The gain is real but modest and uneven: on LIBERO-Plus zero-shot, Language and Light items slightly decrease. After refinement, latent fidelity only improves by SSIM 0.0042 and drops FID 0.01; the authors admit this isn't enough to prove causation, leaving the mechanism to future work. Currently, refinement happens only once at action-block boundaries, with no closed loop within blocks—listed as a future direction.

## Author's Assessment (not from the paper, cross-paper synthesis)

This paper's core claim directly opposes the whole camp of SLIM, JEPA-WAM, and TacWAM—they say world models should only work during training and be fully removed at deployment; this paper says inference still needs one step, just compressed to the cheapest form. Its own ablation provides hard evidence: removing the imagination branch makes the refiner worse than no refinement, not equal.

But the other half of its design sides with the opponents: on decoupling current representation from candidate actions, it does the cleanest validation in the library—this "structurally isolating auxiliary signals" approach shares the same design lineage as TacWAM and JEPA-WAM's shared backbone, just with a different isolation target: they isolate the entire future branch, while this paper isolates only the current state, leaving an extremely lightweight imagination channel. The debate isn't whether to have world models, but to what layer to isolate.

Another recurring pattern: fidelity metrics only improve slightly after refinement, and the authors hesitate to draw causal conclusions—this is the third time after SLIM and omega-0 that "better prediction metrics" and "actually better control" can't be equated.

Anchor for review in six months: if within-block closed-loop refinement is achieved, can it amplify the current modest gains of 0.7 to 2.2 points to an order-of-magnitude level; if not, it suggests the ceiling for a single ultra-lightweight imagination step is around here.