# Opening the VLA Black Box: Progress Signals

Opening the VLA Black Box: Progress Signals

Full article: https://haiguangboy.com/posts/decoding_task_progress_from_vla_representations

## Core Method

Break "represented" into three layers, and test the task progress signal layer by layer
↳ Layer one, weakly decodable—can an in-domain linear probe read out the signal. Use this layer to test task progress (the normalized proportion of remaining trajectory time); the naive probe's in-domain goodness of fit reaches as high as 0.95, which looks like a clean pass
↳ Layer two, strongly decodable—switch to a counterfactual input (replace "pick up the pear" in the instruction with "pick up the apple"), and the same probe, without retraining, must still give the correct response, ruling out "it was actually taking a visual shortcut." The 0.95 probe from layer one reveals its true colors here: swap the instruction and the output barely changes, showing it wasn't using the language input at all and was just guessing right purely from vision; retraining with a contrastive objective fixes this problem, but at the cost of in-domain goodness of fit dropping to 0.33—language sensitivity doesn't come for free
↳ Layer three, steerable—directly intervene on the activations to inject a target progress value, and test whether the model's output actions actually change in that direction. Even with the fixed, strongly decodable probe in hand, injecting an offset of "already advanced 0.2 progress" still doesn't make the model's readings in subsequent layers follow along, and the generated actions don't move in the right direction either—a signal that passed the first two layers fails completely at this layer

## Key Results

· Among the four compared models, three π-series variants perform similarly on the progress feature and clearly outperform the pure vision-language backbone—indicating this feature is established by large-scale robot pretraining, not shaped by task fine-tuning
· To make the probe sensitive to language counterfactuals, you have to retrain it with a contrastive objective at deeper layers, at the cost of in-domain goodness of fit dropping from 0.95 to 0.33—language sensitivity isn't obtained for free
· Using this probe directly as an unsupervised OOD detector matches or even surpasses supervised baselines that require dedicated OOD labels for training, and is actually more robust when facing perturbation types never seen during training

## Related Line Comparison

- Same line · [Gemini_Robotics_2_Safety](https://haiguangboy.com/posts/gemini_robotics_2_safety) `gemini_robotics_2_safety_2026_07`: Core finding: task progress (the normalized proportion of remaining trajectory time) is both weakly and strongly decodable in π0.5's residual stream, but not steerable—injecting an offset of 'already advanced 0.2 progress' along the probe direction does not correspondingly change the model's progress readings in subsequent layers, and the distance between the injected action and the true counterfactual action remains large validates empirical finding: there is a clear gap between the model "understanding constraints" and "acting according to constraints"
- Same line · [LAWM: Why Action Labels Become a Burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why Action Labels Become a Burden
- Same line · [PAVE: Failure Data Can Also Feed Policies](https://haiguangboy.com/posts/pave_predictive_alignment_and_value_guided_evolution_for_world_action_policies) `pave_predictive_alignment_and_value_guided_evolution_for_world_action_policies_2026_09`: PAVE: Failure Data Can Also Feed Policies
- Same line · [PointWorld: Point Flow Unifies State and Action](https://haiguangboy.com/posts/pointworld_scaling_3d_world_models_for_in_the_wild_robotic_manipulation) `pointworld_scaling_3d_world_models_for_in_the_wild_robotic_manipulation_2026_08`: PointWorld: Point Flow Unifies State and Action
- Same line · [The World Model Doesn't Listen Because the Latent Action Is Contaminated](https://haiguangboy.com/posts/cd-lam) `causally_debiased_latent_action_model_for_embodied_action_conditioned_world_mode_2026_07`: A naively trained progress probe achieves in-domain goodness of fit as high as 0.95 (which looks very successful), but under a language counterfactual test (replacing the target object in the instruction) the output barely changes—exposing that this probe actually relies entirely on visual cues and isn't using the language input at all, and that the in-domain high-accuracy metric itself completely failed to catch this problem validates that pixel metrics cannot rank the quality of latent action spaces—this is the most transferable methodological claim in the whole paper
- Same line · [Others Stack Sensors, This One Strips Them All](https://haiguangboy.com/posts/blind_dexterity_whole_body_humanoid_manipulation_via_pure_proprioception) `blind_dexterity_whole_body_humanoid_manipulation_via_pure_proprioception_2026_09`: Others Stack Sensors, This One Strips Them All

## Boundaries

· All experiments use only one model family and are done only in simulation; whether this transfers to other VLA architectures or real robots remains an open question
· The ground-truth label for progress is the time proportion in successful demonstrations, which conflates "time passing" with "the task actually advancing"
· The "not steerable" conclusion only tests a single layer, a single direction, and a fixed magnitude of intervention; the authors explicitly say that stronger interventions or jointly editing multiple locations may yield different results

## Author's Judgment (not part of the paper; a cross-paper synthesis view)

The judgment that "being readable doesn't mean it's actually used for decision-making" is repeatedly corroborated by several works already interpreted: What/Where/How found that video models can organize camera motion into coherent geometric trajectories, yet fail to learn genuine physical reasoning; Blind Dexterity found that feeding a state estimator's predictions back into observations yields no stable benefit for control, but the estimator's own error does systematically decrease after contact, proving the information genuinely exists. Three papers hit the same wall from three different angles: a signal can be stably read out without being the variable the model uses to act.