# Agile-WAM: Lightweight Visuotactile Joint Prediction

Agile-WAM: Lightweight Visuotactile Joint Prediction

Full article: https://haiguangboy.com/posts/agile-wam

## Core Method

Lightweight joint prediction, where vision and touch each look at a different future
↳ ① Current observations directly drive action generation
Vision, touch, and the robot's own state are encoded into a shared latent variable, which serves as the starting point for flow matching; a lightweight MLP jointly generates actions, future visual latents, and future tactile latents in one pass. Observations are encoded only once, and subsequent integration does not repeatedly inject conditions, nor does it generate future video pixels.

↳ ② Vision looks far, touch looks near
Each generation produces 16 action steps, executing the first 8 before replanning. Vision predicts step 8, avoiding adjacent frames being too similar and supervision being too easy; touch predicts only the next step, preserving contact discontinuities. In ablations, having both modalities predict the next step, or both predict a more distant future, degrades performance.

↳ ③ Predictions must also translate into executable actions
During training, the generated action latents are decoded and directly supervised, reducing the gap between training and inference. Keeping the same latent space size but turning off only the future prediction loss still degrades performance: the gains cannot be explained by increased capacity alone.

## Key Results

· 9 simulation and 5 real-robot tasks; 50 demonstrations and 20 tests per real-robot task. Gear assembly success rate 80%, power plug insertion 65%, versus VITA-VT's 55% and 35% respectively; however, peg insertion was 55%, below the baseline's 60%, and network cable insertion was only 30%.
· Generating 16 action steps takes about 10.35 ms on an RTX4090, close to VITA's 9.71 ms; low latency is maintained even after adding touch and future prediction.

## Comparison with Related Lines of Work

- Opposing line · [WAM survey: how prediction serves control](https://haiguangboy.com/posts/wam-survey) `world_action_models_for_robot_learning_and_control_a_survey_2026_09`: keeping a large latent space but removing the prediction loss still degrades performance contradicts the idea that world prediction and action selection can be optimized separately
- Opposing line · [Touch as a prediction target rather than an observation input](https://haiguangboy.com/posts/n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens) `n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens_2026_07`: touch as a prediction target rather than an observation input
- Same line · [JEPA-WAM: an innovative method that helps world models better understand state changes](https://haiguangboy.com/posts/jepa-wam) `jepa_wam_learning_vision_language_action_policies_with_joint_embedding_world_modeling_2026_08`: keeping a large latent space but removing the prediction loss still degrades performance validates five ablation groups: joint objective 79.2 versus future-only objective 77.3, preserving spatial structure versus iREPA convolution 74.7
- Same line · [N0-Foundation: opening a new era of touch](https://haiguangboy.com/posts/mathcaln_0_foundation_towards_the_age_of_tactile_intelligence) `mathcaln_0_foundation_towards_the_age_of_tactile_intelligence_2026_09`: N0-Foundation: opening a new era of touch
- Same line · [PRISM: can scale replace explicit modeling?](https://haiguangboy.com/posts/prism_precision_and_contact_rich_real_world_industrial_skill_dataset_with_multim) `prism_precision_and_contact_rich_real_world_industrial_skill_dataset_with_multim_2026_08`: PRISM: can scale replace explicit modeling?
- Same line · [SmoothRL: only accounting for actions actually executed](https://haiguangboy.com/posts/smoothrl_online_reinforcement_learning_during_asynchronous_execution) `smoothrl_online_reinforcement_learning_during_asynchronous_execution_2026_09`: SmoothRL: only accounting for actions actually executed

## Boundaries

⚠️ Boundaries
About 97 Hz is only the inference capability; real-robot vision and touch sampling are both 30 Hz, so deployment still requires verifying the full latency of sensing, inference, and execution. Corrective trajectories were actively added to the demonstrations, so it cannot be said that the model learned error correction from scratch; the visual encoder also used ImageNet pretraining.

## Author's Assessment (not part of the paper; a cross-paper synthesis view)

Viewed alongside T-Rex, the difference is quite concrete: T-Rex splits slow visual and fast tactile correction pathways during execution; Agile-WAM distinguishes visuotactile prediction horizons during training. Both value timescales, but "next-step tactile prediction" is not the same as "high-frequency tactile closed loop."
What is worth rechecking from an engineering standpoint is: after changing tasks or sampling rates, does setting separate prediction horizons still outperform a unified horizon? This determines whether it can become a reusable method, rather than merely an advantage under the current configuration.