# RL Should Not Be Learned from Scratch; It Should Be Post-Trained

RL Should Not Be Learned from Scratch; It Should Be Post-Trained

Full article: https://haiguangboy.com/posts/rl-100

## Core Method

Imitation learning provides a foundation, offering a physically feasible behavioral prior; iterative offline RL, under conservative updates paired with progressive data expansion, extracts most of the performance; brief online fine-tuning only handles rare residual failures. All three stages share the same clipped PPO objective, so transitioning from offline to online does not require changing the learning primitive.

Key design: diffusion denoising is structured as a two-level MDP, with the same environment-level advantage shared across all denoising steps, addressing the weak credit assignment from scoring only the final action. The offline stage also includes a gate—candidate policies are advanced only if conservatively predicted to yield genuine improvement; otherwise, they are rejected, preventing sudden performance collapse.

## Key Results

Eight real-world tasks, 1,000 trials, 100% success. The offline RL stage contributes most of the improvement from the imitation baseline to a 91.8% average, while the online stage only cleans up residuals. In terms of data budget, human demonstrations average just 1.8 hours per task, accounting for less than 13% of the total collection budget; the rest comes from autonomous policy rollouts. At deployment, it is distilled into a one-step consistency policy, running at 378Hz inference with only 3.9M parameters.

On Push-T, the number of completions per unit time exceeds expert teleoperators by 1.18 times and novices by 1.54 times.

## Comparison with Related Approaches

- Same approach · [1 day, 1,000 tasks, relying on inductive biases](https://haiguangboy.com/posts/mt3-thousand-tasks) `learning_a_thousand_tasks_in_a_day_2026_08`: Data budget: human demonstrations average only 115 episodes per task (1.8 hours), accounting for less than 13% of the total data collection budget, validating the main controlled experiment results: MT3 outperforms other methods with 3 demonstrations versus their 50.
- Same approach · [LAWM: Why action labels become a burden](https://haiguangboy.com/posts/latent_action_pretraining_through_world_modeling) `latent_action_pretraining_through_world_modeling_2026_07`: LAWM: Why action labels become a burden
- Same approach · [At which level should RL intervene](https://haiguangboy.com/posts/zprl) `beyond_action_residuals_real_world_robot_policy_steering_via_bottleneck_latent_r_2026_08`: The offline stage uses an approximate model Q-function as a gate, advancing the behavior policy only when improvement is predicted; otherwise, the update is rejected, validating the core claim: the choice of intervention level is itself a key design variable, not an implementation detail.
- Same approach · [π0.5 shudders in place when failing to grab a spoon, but Orca goes further with physical intuition learned from watching videos](https://haiguangboy.com/posts/orca) `orca_2026_07`: The key to a world model is readable states
- Same approach · [Switching logic should not be left for the policy to learn](https://haiguangboy.com/posts/dr-lfd) `decompose_and_reorganize_planning_with_primitives_and_visuomotor_policies_learne_2026_08`: Data budget: human demonstrations average only 115 episodes per task (1.8 hours), accounting for less than 13% of the total data collection budget, validating results: 100% on simulated peg-in-hole ID vs ACT 44%/DP 54%; on DexMimicGen, 100 demonstrations outperform the baseline's 1,000.
- Same approach · [No need to imagine the future at inference time—robots still reach 91.8%! Fast-WAM debunks WAM's core assumption](https://haiguangboy.com/posts/fast-wam) `fast-wam_2026_07`: Training video objectives matter more than imagining the future at test time

## Limitations

In the juice extraction task, the pressing step is a fixed trajectory recorded via kinesthetic teaching and replayed at deployment—it is not learned—so strictly speaking, a small segment within the "eight tasks" is scripted. The authors themselves acknowledge that autonomous reset remains an unresolved bottleneck and list it as future work. The abstract also notes that the online exploration phase involves "gentle human guidance," not full autonomy.

## Author's Assessment (Not part of the paper; cross-paper synthesis)

This paper's core claim—that RL should be post-trained around deployment metrics rather than learned from scratch or purely approximating demonstrations—is independently corroborated by two other findings. One is simulation-first, with pretraining barely touching real robots, followed by a small amount of real-world RL after deployment; the other inverts the data pyramid, using human data as the foundation rather than internet videos. The three approaches start from different points but converge to the same shape: a strong prior plus lightweight real-world post-training.

That offline gate also connects to another work just completed—its conclusion is that the level at which RL should intervene is itself a design variable as important as how much to modify. The gating mechanism in this paper is a concrete implementation of that "intervention level choice."