# SmoothRL: Only settle accounts for actions that are actually executed SmoothRL: Only settle accounts for actions that are actually executed Full article: https://haiguangboy.com/posts/smoothrl_online_reinforcement_learning_during_asynchronous_execution ## Core method Under asynchronous execution, the actions generated by the policy and the actions actually executed by the robot are two different things—compute gradients only for the segment that is actually executed ↳ Each action chunk is divided into three segments according to execution status: the committed segment (already covered by the previous chunk long ago and will never reach the environment), the execution segment (the only part truly sent to the robot and directly determining the trajectory outcome), and the discarded segment (replaced by the next chunk before it is executed). A fixed delay budget turns the boundaries of these three segments into constants uniquely determined by frame indices ↳ The value gradient is computed only for the execution segment—because this segment of actions is simultaneously the true output of the current policy and the action actually executed by the robot, and this identity relationship makes the gradient well-defined with respect to the policy parameters. However, the critic's input is still the complete action chunk, used to ensure unbiased cross-chunk bootstrapping and to model the concurrent decision process in which "the previous chunk is still in flight" ↳ The asynchronous loop runs as usual during training rollouts, not only during deployment—so the replay data records the actions actually executed under a timing structure fully consistent with the objective function, and the policy is never optimized under conditions it would not encounter at deployment ## Key results · On three real-robot tasks, after 250 rounds of online RL there were large improvements: dynamic throwing 39%→94%, pen capping 8%→83%, box opening 30%→90%—what was corrected were systematic fixed biases (the blade consistently drifting left, release speed not coupled to distance), not random noise · The raw action space allows human interventions to be used directly as training data without conversion: residual-style intervention (preserving the base velocity profile and only correcting deviations) achieves about 80% success on the hardest configuration, far exceeding the about 30% of direct takeover · Smoothness is not automatically guaranteed—once the value objective starts pushing Q-values higher, velocity/acceleration/jerk constraints must be explicitly added back into the optimization objective ## Comparison with related lines of work - Opposite line · [RL should not learn from scratch, it should do post-training](https://haiguangboy.com/posts/rl-100) `rl_100_performant_robotic_manipulation_with_real_world_reinforcement_learning_2026_08`: On three real-robot tasks, after 250 rounds of rollout online reinforcement learning, all improved substantially relative to the frozen base policy: dynamic throwing from 39% to 94%, pen capping from 8% to 83%, box opening from 30% to 90%; the base policy's failures are systematic fixed biases (not random errors), and online RL mainly corrects these systematic biases contradicts 100% success across 1000/1000 trials on eight real-robot tasks; the offline RL stage contributed most of the improvement, while the online stage only cleaned up residual... - Opposite line · [At which layer should RL intervene](https://haiguangboy.com/posts/zprl) `beyond_action_residuals_real_world_robot_policy_steering_via_bottleneck_latent_r_2026_08`: The smoothness of the pretrained policy comes from demonstration data, and once the value objective starts pushing Q-values higher on the execution segment, this smoothness guarantee no longer automatically holds—smoothness must be added back into the optimization objective as an explicit constraint (boundary penalty terms for per-frame velocity/acceleration/jerk) contradicts Mechanism explanation: steering is remapping state-action associations, not inventing new actions - Opposite line · [Tactile as prediction target rather than observation input](https://haiguangboy.com/posts/n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens) `n_0_vtla_scaling_vision_tactile_language_action_model_with_latent_tactile_tokens_2026_07`: Tactile as prediction target rather than observation input - Same line · [An_Open_Foundation_Model_Towards](https://haiguangboy.com/posts/an_open_foundation_model_towards) `an_open_foundation_model_towards_2026_07`: The asynchronous inference loop actually runs during the training-time rollout process, not only during deployment—this way the replay data records the actions the robot actually executed under a timing and execution schedule fully consistent with the objective function, ensuring the policy is never optimized under dynamics conditions it would not encounter at deployment validates RTC at training time: masking the first d action tokens so the model learns smooth continuation - Same line · [N0-Foundation: Opening a new era of tactile intelligence](https://haiguangboy.com/posts/mathcaln_0_foundation_towards_the_age_of_tactile_intelligence) `mathcaln_0_foundation_towards_the_age_of_tactile_intelligence_2026_09`: N0-Foundation: Opening a new era of tactile intelligence - Same line · [VLA action chunk switching no longer stutters](https://haiguangboy.com/posts/learning_native_continuation_for_action_chunking_flow_policies) `learning_native_continuation_for_action_chunking_flow_policies_2026_09`: VLA action chunk switching no longer stutters ## Boundaries · It depends on a fixed inference delay budget; once actual fluctuations exceed the budget, the timing of the entire asynchronous loop will be disrupted · The expressive capacity of the RL module is doubly locked down by the frozen base: the perception ceiling is constrained by the amount of information in the VLA's internal representations, and the correction magnitude is limited to a local neighborhood around the reference actions · The real-robot evaluation scale is relatively small (10-18 configurations each run once, with success manually judged), characterizing results under this specific setup rather than large-scale statistical conclusions ## Author's judgment (not part of the paper; a cross-paper synthetic view) This paper and the previously interpreted RL-100 form a divergence worth noting: RL-100's own finding is that the offline RL stage contributed most of the improvement, while the online stage only cleaned up residual failures; this paper instead treats online RL as the core means of correcting systematic biases (not residual noise). The two sides give different answers to "how heavy a load online RL should carry," which may depend on how close the pretraining stage itself brings the policy to success. In addition, this paper's principle of "also running the asynchronous loop during training so that the optimization objective aligns with the real timing at deployment" is in the same direction as the recently interpreted Legato—both papers emphasize that the training stage must never turn a blind eye to the timing structure encountered at deployment.