# VGEBench: Generalizable Visual Exploration

VGEBench: Generalizable Visual Exploration

Full article: https://haiguangboy.com/posts/towards_generalizable_visually_grounded_exploration_of_household_devices

## Core Method

968 3D models across 26 device categories, no manuals provided, relying purely on visual perception and interaction feedback to infer operational logic
↳ Each device is instantiated as a concrete state machine, precisely defining which component corresponds to which action and what state transitions are triggered; the agent can only observe images, select actions, and see feedback at each step. The system performs hierarchical validation based on whether coordinates fall within the correct component bounding box and whether action type and parameters match transition conditions, judging success by whether goal states are achieved in sequence without mandating a specific path
↳ 14,953 task episodes, of which 10,005 involve multi-turn interactions, specifically testing long-horizon exploration reasoning

## Key Results

· Strongest model Gemini-3-Flash achieves 54.27% success rate, second place drops sharply to 16.68%, and all other models fall below 15%
· Counterexample: MiMo-Embodied-7B has the highest navigation efficiency (0.88, surpassing Gemini-3-Flash), yet its task success rate is only 1.48%—the real bottleneck is precisely locating interactive components, not finding the right viewpoint (localization error gap is 63.9%, navigation gap only 12.5%)
· Even for the strongest model, the "get it right on the first try" rate is only 12.94%, but the success rate with trial-and-error allowed reaches 62.86%—relying on repeated exploration rather than one-shot understanding

## Related Work Comparison

- Same direction · [1 Day 1000 Tasks, Relying on Inductive Biases](https://haiguangboy.com/posts/mt3-thousand-tasks) `learning_a_thousand_tasks_in_a_day_2026_08`: Author's judgment: navigation exploration ability and fine-grained visual localization ability are separable capabilities; the former being strong does not imply the latter is strong, and task success in this domain is determined by the latter. Validates perception dependence: vision-only, single camera, no tactile feedback, relies on accurate segmentation
- Same direction · [N0-Foundation: Towards the Age of Tactile Intelligence](https://haiguangboy.com/posts/mathcaln_0_foundation_towards_the_age_of_tactile_intelligence) `mathcaln_0_foundation_towards_the_age_of_tactile_intelligence_2026_09`: N0-Foundation: Towards the Age of Tactile Intelligence
- Same direction · [Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design](https://haiguangboy.com/posts/transformer-transformer-codesign) `transformer-transformer_blog_transformer_transformer_a_unified_20260807_2026_08`: Core positioning judgment: manual-free generalizable visually grounded exploration is a critical capability gap in current VLM agents, not covered by traditional imitation/reinforcement learning robot benchmarks, nor by LLM tool-calling paradigms relying on explicit APIs/manuals. Validates input still consists of target end-effector trajectories and hand-crafted rewards, not direct task understanding from language and scenes
- Same direction · [Liang Wenfeng's Four-Hour Investor Meeting Transcript](https://haiguangboy.com/posts/liangwenfeng-world-model) `wx_elsewhere别处发生_20260722_2026_07`: Core positioning judgment: manual-free generalizable visually grounded exploration is a critical capability gap in current VLM agents, not covered by traditional imitation/reinforcement learning robot benchmarks, nor by LLM tool-calling paradigms relying on explicit APIs/manuals. Validates ★★Core judgment: the implicit premise of "world models are unrelated to intelligence ceilings" is that "data pipelines are already connected"—for LLMs they are, so it's a detour; for robots they aren't, so it's a bridge
- Same direction · [Latent Actions: Optical Flow as a Liability](https://haiguangboy.com/posts/what_matters_for_latent_actions_in_robot_learning) `what_matters_for_latent_actions_in_robot_learning_2026_08`: Latent Actions: Optical Flow as a Liability
- Same direction · [Neural Signals: Their Value and Fragility Stem from the Same Source](https://haiguangboy.com/posts/real_time_semg_based_telecontrol_of_an_assistive_robotic_arm_using_a_1d_convolut) `real_time_semg_based_telecontrol_of_an_assistive_robotic_arm_using_a_1d_convolut_2026_07`: Neural Signals: Their Value and Fragility Stem from the Same Source

## Limitations

· This is an abstract simulation with rendered views, discrete actions, and fixed viewpoints, not continuous control of real robots, and does not fully cover sim-to-real challenges in physical deployment
· Observations are clean, centered device renderings without real backgrounds, lighting, or occlusions; perturbation tests cannot fully substitute real scenes
· 3D assets come from public websites, and the authors acknowledge they cannot rule out that closed-source models may have encountered related assets during pretraining

## Author's Judgment (Beyond Paper Content, Cross-Paper Synthesis)

The most intriguing thread: the judgment that "giving systems more information or longer context is not unconditionally beneficial" has been independently confirmed across completely different domains. This paper finds that longer interaction histories only help strong models, while weak models perform better with shorter histories—because weak models cannot distinguish useful feedback from historical noise. This aligns with several previously reviewed works: in What Matters for Latent Actions, more motion-aware optical flow representations actually hurt performance; TacWAM shows that making auxiliary modalities "visible" causes greater loss than missing information; FLEX-π emphasizes that the key mechanism is forcing reconstruction of missing flow itself, not just "seeing more modalities." Across four different fields—language models, tactile representations, latent actions, and VLM agents—the same judgment recurs: whether the receiver can filter and integrate information matters more than how much information is provided. Additionally, this paper's positioning of "manual-free exploration capability" echoes another broader judgment—that world models are more needed in robotics than in language models, precisely because robots' "data pipelines" are not yet connected, requiring an additional bridge rather than a detour as with LLMs.