# Latent Actions: Optical Flow Is a Liability

Latent Actions: Optical Flow Is a Liability

Full article: https://haiguangboy.com/posts/what_matters_for_latent_actions_in_robot_learning

## Core Method

↳ Ran through 41 design choices; among the seven modeling paradigms, the simplest LAPO (no preprocessing, trained directly on raw data) scored highest overall (0.7327); optical flow methods consistently ranked last, failing to beat even pixel-difference baselines, and adding optical flow constraints to LAPO made it worse
↳ Reason: optical flow converts visual changes into explicit pixel displacements, and this "motion-centric" inductive bias loses contact, occlusion, and local deformation cues tied to transition dynamics, while injecting estimation errors into the representation—more structured priors don't necessarily mean better representations
↳ Using pretrained DINO features for semantic differences approaches or even surpasses LAPO: a strong semantic prior is itself an effective inductive bias, no extra motion structure needed

## Key Results

· Regularization method type has limited impact; strength is what matters: VAE recommends 10⁻⁷, Sparsity recommends 10⁻⁵, SIGReg recommends 10⁻³, VQ-VAE uses 1
· Latent action dimension 32 is the universal sweet spot for single-arm and dual-arm setups; with proper regularization, skipping normalization performs better (28/33 combinations improved)
· Real-robot validation: VLM backbones fine-tuned with latent actions on Franka Panda improved success rates from 64.75% to 79.25%, with 4x faster convergence

## Related Route Comparisons

- Opposite route · [PointWorld: Point Flow Unifies State and Action](https://haiguangboy.com/posts/pointworld_scaling_3d_world_models_for_in_the_wild_robotic_manipulation) `pointworld_scaling_3d_world_models_for_in_the_wild_robotic_manipulation_2026_08`: PointWorld: Point Flow Unifies State and Action
- Same route · [LAWM: Why Action Labels Become a Burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why Action Labels Become a Burden
- Same route · [Where Should RL Intervene](https://haiguangboy.com/posts/zprl) `beyond_action_residuals_real_world_robot_policy_steering_via_bottleneck_latent_r_2026_08`: Using pretrained DINO features for semantic differences (DeltaDINO) approaches or even surpasses LAPO, being the best overall on LIBERO validates the core claim: the choice of intervention point is itself a key design variable, not an implementation detail
- Same route · [It's Not "Whether to Use World Models," It's "Whether What They Output Has Structure"](https://haiguangboy.com/posts/robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode) `robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode_2026_07`: It's Not "Whether to Use World Models," It's "Whether What They Output Has Structure"
- Same route · [World Models Misbehave Because Latent Actions Are Contaminated](https://haiguangboy.com/posts/cd-lam) `causally_debiased_latent_action_model_for_embodied_action_conditioned_world_mode_2026_07`: All proxy metrics (including the FDM reconstruction metric the paper itself deems most reliable) can only serve as coarse filters, not substitutes for real downstream task evaluation—a warning for the entire LAM research methodology validates evaluation only on offline rollouts of 300 clips each from EgoDex/AgiBot, with no closed-loop task success rates; training requires 96 H100s
- Same route · [World Models: Whether to Imagine at Inference Time](https://haiguangboy.com/posts/omega-eva) `omega_eva_envision_verify_and_act_with_latent_interactive_world_models_2026_08`: Core design principle: latent actions shouldn't just be auxiliary supervision signals during pretraining; they must stay engaged throughout downstream policy learning to maximize value validates problem breakdown: existing world model uses fall into three categories, none of which let candidate actions be truly tested and corrected by their own imagined outcomes

## Boundaries

These rankings don't represent deployment-optimal choices—evaluation uses only front views, omits joint states and data augmentation, a clean-room setup to exclude confounders; findings on 14-DoF dual-arm setups (like dimension 32) are only validated in simulation, without real-robot data support; latent actions are currently task-specific signals at the fine-tuning stage, not yet elevated to foundation-level representations

## Author's Judgment (Not Paper Content, Cross-Paper Synthesis)

The sharpest takeaway here: all proxy metrics can only do coarse filtering, can't produce fine-grained rankings, and can't replace real downstream evaluation. This echoes two previously reviewed papers—Omega-0 found models with higher offline reconstruction quality actually executed more sluggishly on real robots; CDLAM reached the same conclusion, that pixel-level reconstruction metrics can't rank latent action spaces. But intriguingly, this directly clashes with the just-reviewed PointWorld: PointWorld deliberately abandons task success rates and adopts per-point L2 error as its core metric, arguing that "success rates mask systematic differences"—one says no matter how fine-grained, proxy metrics are unreliable, the other says switching to a finer metric reveals gaps; the answers are opposite, and which holds up needs more evidence.