← Projects

Evaluating Robot Policies in a Visuo-Tactile World Model

Current video world models [1, 2, 3] act as effective physical simulators to evaluate robot policies. However, the real world is inherently multimodal: aside from vision, humans learn to interact with their environment through audio, tactile, and internal force feedback. In contact-rich manipulation, visuo-tactile policies have shown clear improvements over vision-only policies [4].

How can we train multimodal world models for robots to learn visuo-tactile policies in simulation?

classical simulator diffused pixels
World model jointly generating front, wrist, tactile RGB, and tactile force field vs the classical simulator.
front viewwrist view tactile RGBtactile force field

Simulator

Contact-rich manipulation is simulated in NVIDIA Isaac Gym using the ManiFeel [5] task suite with TacSL, which renders GelSight optical-tactile images via Taxim and per-taxel shear force fields at each contact.

World model

We train a multimodal autoregressive Diffusion Forcing DiT. It is trained on demonstration rollouts collected in the simulator above, where every frame pairs the four observation streams with the robot's action delta at that step. The figure above is then produced by conditioning on a single starting frame and the demonstration's action sequence and autoregressively rolling out all four modalities on a held-out episode. Each frame's front, wrist, and tactile-RGB latents plus the raw tactile force field are tokenized into a single sequence and jointly diffused. Generating the front and wrist camera views jointly encourages a consistent underlying scene across viewpoints. Every DiT block interleaves spatial attention with temporal attention. Each modality carries its own per-group 2D rotary position embedding so heterogeneous grids stay correctly positioned. Rather than treating the tactile force field as an image, the raw force field is co-diffused directly through a small parallel low-dimensional head, and drawn as shear arrows only for display.

References

  1. [1] Quevedo et al. WorldGym: World Model as an Environment for Policy Evaluation, 2025.
  2. [2] Sharma et al. World-Gymnast: Training Robots with Reinforcement Learning in a World Model, 2026.
  3. [3] Gemini Robotics Team. Evaluating Gemini Robotics Policies in a Veo World Simulator, 2025.
  4. [4] Huang et al. 3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing, 2024.
  5. [5] Luu et al. ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning, 2025.