# At which layer should RL intervene

At which layer should RL intervene

Full article: https://haiguangboy.com/posts/zprl

## Core method

Offline, attach a variational information bottleneck module to the base model to extract compact task-relevant latent variables; online, freeze the base model, and RL only learns residual perturbations applied to these latent variables, which are transformed into new conditions via the frozen decoder, then actions are generated by the frozen policy.

There is also an easily overlooked design: the critic is defined on the perturbed latent variables rather than the residuals themselves—the same residual added to different latent variables induces different action distributions, and without a base, it cannot be priced.

## Key results

Across four real-world tasks, after several hours of online interaction, the average success rate improved by 33.7%, with card insertion rising from 20% to 77.5%. A byproduct is that actions remain smooth during online exploration, whereas the action residual method becomes increasingly oscillatory with training.

## Comparison of related approaches

- Opposite approach · [Noise in teleoperation data must enter the loss function](https://haiguangboy.com/posts/dexora) `dexora_open_source_vla_for_high_dof_bimanual_dexterity_2026_08`: Simulation: all three benchmarks and eight tasks reach strong final performance, covering parallel grippers and dexterous hands contradicts the simulation-to-dexterity gap: simulation can guide basic skills, but dexterity must rely on real-world data
- Same approach · [It's not "whether to have a world model," it's "whether what it outputs has structure"](https://haiguangboy.com/posts/robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode) `robointer15_a_holistic_intermediate_representation_suite_for_embodied_world_mode_2026_07`: It's not "whether to have a world model," it's "whether what it outputs has structure"
- Same approach · [World models misbehave because latent actions are contaminated](https://haiguangboy.com/posts/cd-lam) `causally_debiased_latent_action_model_for_embodied_action_conditioned_world_mode_2026_07`: Key ablation: the gain does not simply come from dimensionality reduction—dim(z)=64 already exceeds the action dimension yet is still faster and better validates that scale cannot replace debiasing: baselines from 2B to 14B improve vision but worsen action following
- Same approach · [Glove tactile readings are mixed with hand pose](https://haiguangboy.com/posts/tactile-glove-pose-artifacts) `pose_aware_modeling_to_mitigate_pose_related_artifacts_in_tactile_gloves_2026_07`: Controlled experiment: residual on observation embeddings (RESEMB) is consistently worse and becomes more unstable as residual scale increases validates again that 'direct concatenation is worse': this time the explanation is that the model learns the wrong mapping direction
- Same approach · [The bottleneck in robot RL is sampling, not algorithms](https://haiguangboy.com/posts/pi-rl-chelsea-finn-talk) `x_manual_pi_rl_20260726_20260727_2026_07`: Online cost is not low: real-world tasks require hours of interaction, and λ must be manually determined per task validates per report: the bottleneck for RL for robotics is the cost of physical rollouts, not just algorithm quality
- Same approach · [The overlooked non-consensus is whole-body unified control](https://haiguangboy.com/posts/gemini-robotics-2) `deepmind_blog_gemini_robotics_2_brings_20260730_2026_07`: Simulation: all three benchmarks and eight tasks reach strong final performance, covering parallel grippers and dexterous hands validates dexterity: simultaneously covers a 22-DOF five-finger hand and a standard two-finger gripper

## Boundaries

Performance is limited by the base model's support domain—reducing offline demonstrations from 100 to 25 noticeably degrades online performance. Robustness holds only under moderate perturbations, averaging 69%, and the deformable banknote task worsens after perturbation, with only 10 trials per case. Cost is also high: real-world online interaction time is three to four times that of offline collection, and the perturbation scale must be manually calibrated per task (0.1 to 1.5).

## Author's judgment (not from the paper, synthesized across papers)

This paper converges with another judgment: auxiliary signals cannot be plugged into the main trunk without constraints; they need a structurally restricted position. Previous methods used residual terms, prediction targets, dedicated pathways, and attention masks; this one uses "adding perturbations in a compressed task latent space." The increment lies in three sets of cross-evidence that isolate "not dimensionality reduction," "not arbitrary features," and "must be local."

There is also a fresh divergence. A few days ago, the bimanual work concluded that simulation can only guide basic skills, dexterity must rely on real robots, and pure simulation is near zero on dexterous tasks. This paper achieves strong performance with RL fine-tuning in pure simulation on the Adroit dexterous hand. The difference may lie in whether there is a good base model to build on—one learns dexterity from scratch, the other guides on existing priors.