Interactive world simulator for robot policy training and evaluation

Researchers describe an Interactive World Simulator, a learned action-conditioned video-prediction system intended to reduce the cost and reproducibility problems of collecting robot demonstrations and evaluating policies on physical hardware. Rather than using a physics engine, it takes a current image and robot-action sequence and predicts future pixel-space frames. The system can be controlled with teleoperation for more than 10 minutes at 15 frames per second on one RTX 4090 GPU, while its predicted video is reported to remain stable and physically plausible.

Training uses two stages. An autoencoder first compresses RGB images into two-dimensional latent representations and reconstructs images. A frozen autoencoder then supports an action-conditioned latent dynamics model, which uses prior visual latents and actions to predict the next latent state. During inference, decoded predictions are fed back autoregressively as context. The approach uses consistency models to support interactive, long-horizon operation.

The simulator was trained solely from interaction data on T pushing, rope routing, mug grasping, and pile sweeping. Reported behaviors include distinguishing rope insertion into a clip from a non-contact swing, modeling mug slips and handle rotation, and generating consistent multiple views for sweeping.

For data generation, demonstrations collected entirely in the simulator were used to train imitation-learning policies with no real-world training data. When deployed on a real robot, these policies completed tasks and remained robust to continuous human perturbations; the article says their generated-data quality was comparable to real data.

For evaluation, four policies—DP, ACT, π0, and π0.5—were tested in the simulator and on a real robot from matching initial configurations. Policy-checkpoint scores showed strong correlation across tasks and policies. The team plans to broaden the environments and manipulation tasks and study scaling with more interaction data and computation.