# Scale isn't given by the model—it's given by the robot

Using the robot as its own ruler, reconstruction error is 3.31mm—generative models are nearly 5x worse

Full article: https://haiguangboy.com/posts/splatting-physical-scenes

## Core Method

↳ ① SplatMesh: triangular meshes handle geometry, 3DGS handles appearance; Gaussians are pinned to mesh surfaces and deform with the mesh. Spherical meshes start with 642 vertices, only vertices deform, connectivity stays fixed.
↳ ② Surface element constraint: Gaussian covariance aligns with surface normals, normal scale is clamped to float32 machine precision. Removing it drops PSNR from 30.91 to 25.82—unconstrained Gaussians cheat with background colors or near-full transparency.
↳ ③ Differentiable rendering + differentiable physics (MuJoCo MJX) chained together; pixel error gradients backpropagate simultaneously to mesh vertices, camera poses, and joint angles.

## Key Results

· Real-robot YCB geometry error (√CD): blue tuna can 3.31mm, proprioception baseline 13.92, scale-aligned TRELLIS 16.17.
· Calibration: joint angle noise of 0.01 rad reduces tool pose error from 10.9mm to 3.79mm.
· Single-object reconstruction takes ~157 seconds (H100); assets can be baked with textures and directly imported into MuJoCo.

## Related Approach Comparisons

- Opposite approach · [Closes the kinematic loop, not the physical one](https://haiguangboy.com/posts/video2robo) `deng_video2robo_3dgs_based_synthetic_data_from_one_video_enables_scalable_robot_2026_08`: problem setup: directly generate simulation assets from imperfect data of low-cost robots, rather than requiring clean collection conditions first contradicts this is not physical simulation: the entire pipeline is kinematic, with no contact forces, friction, or dynamics models
- Opposite approach · [Predictive features can't be directly fed to diffusion models](https://haiguangboy.com/posts/leapbot-wa) `leapbot_wa_world_anchor_action_models_via_predictive_latent_alignments_2026_07`: two representation hard limits: MJX only supports rigid bodies, 3DGS rendering models can't be relit contradicts problem diagnosis: mainstream WAMs inherit from video generation, treating world modeling as a pixel-level rendering problem
- Same approach · [D4RT_Efficiently_Reconstructing_Dynamic_Scenes_One](https://haiguangboy.com/posts/d4rt_efficiently_reconstructing_dynamic_scenes_one) `d4rt_efficiently_reconstructing_dynamic_scenes_one_2026_06`: key difference from TRELLIS: this method outputs metric scale and 6D poses, generative models don't validates D4RT output is relative/up-to-scale, not metric—this is the fatal flaw as a source for embodied 3D data
- Same approach · [Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design](https://haiguangboy.com/posts/transformer-transformer-codesign) `transformer-transformer_blog_transformer_transformer_a_unified_20260807_2026_08`: two representation hard limits: MJX only supports rigid bodies, 3DGS rendering models can't be relit validates design space lacks complex meshes, scenes, and contact targets; key gaps remain for assembly and dexterous contact
- Same approach · [Teleop data noise needs to go into the loss function](https://haiguangboy.com/posts/dexora) `dexora_open_source_vla_for_high_dof_bimanual_dexterity_2026_08`: approach claim: imprecision in low-cost robots isn't an obstacle to bypass but something to correct in situ with visual feedback validates core engineering claim: teleop data noise must be explicitly modeled, can't be diluted by scale
- Opposite approach · [World models misbehave because latent actions are polluted](https://haiguangboy.com/posts/cd-lam) `causally_debiased_latent_action_model_for_embodied_action_conditioned_world_mode_2026_07`: engineering finding: COLMAP gives no consistent solution on this data, SAM2 can't segment the robot body either contradicts method: three debiasing objectives—ego-centric reconstruction, action-centric contrast, latent space calibration

## Limitations

Physics is integrated, but the paper explicitly states it only covers object reconstruction and kinematics; contact forces and dynamics identification are not addressed. Meshes can only be homeomorphic to the initial topology—starting from a sphere means staying a sphere; objects with holes or handles won't work. MJX only supports rigid bodies, and 3DGS can't be relit, so lighting looks wrong when objects move. Calibration is sim-to-sim with synthetic noise; real-robot PSNR also required additional pose alignment.

## Author's Assessment (not from the paper; cross-paper synthesis)

The previous Video2Robo bet was to bypass physics with kinematics plus photorealistic rendering; this paper goes the opposite way. Interestingly, the same judgment pulls two papers in opposite directions: LeapBot-WA argues the utility of world modeling lies in abstract physical predictions rather than photorealistic rendering—Video2Robo contradicts it, while this paper corroborates it.

The scale thread ties it together. D4RT's output is up-to-scale, not metric; Video2Robo relies on VGGT's monocular depth to fill in; this paper integrates MJX kinematics into the optimization loop, and the robot's own mechanical dimensions are the ruler—so it dares to claim metrically accurate, whereas TRELLIS doesn't even provide scale or pose.

But both papers share the same ceiling: they only support rigid bodies.

Anchor for review in six months: calibration error of 0.01 rad can be corrected to 3.79mm, but 0.03 rad only reaches 18.5mm—the authors don't discuss this degradation. Whether contact-heavy tasks can tolerate residual errors of one to two centimeters is the watershed for whether this approach can truly serve as a physical foundation.