AI agents create virtual playgrounds to help robots get crucial training data

MIT CSAIL and Toyota Research Institute researchers developed SceneSmith, a system that uses three vision-language-model agents to build simulation-ready 3D indoor environments from text prompts. The project addresses a robotics data bottleneck: teaching machines many tasks in varied real settings is slow and labor-intensive, while simulations need richer content to represent everyday environments.

SceneSmith assigns separate roles to a designer, critic, and orchestrator. The designer proposes layouts and objects, the critic checks whether each stage is realistic and practical, and the orchestrator decides when to approve work or revisit earlier choices. The process builds floor plans, furniture, wall and ceiling items, and manipulable objects, including articulated cabinets. Physics software then adds physical mechanics. The system can also create individual objects from generated images and assign properties such as mass, friction, and inertia.

Researchers produced more than 1,300 scenes, including restaurants, bedrooms, hotels, offices, stores, and a gaming room. They report that SceneSmith scenes contained up to six times as many items as previous methods. In comparisons with HSM and Holodeck, more than 200 users judged its visuals more realistic over 90% of the time and found that it generally followed prompts more closely.

The team generated 100 unique spaces to test robot action plans. A VLM evaluator identified faulty plans, and human reviewers agreed with its judgments more than 99% of the time. A robot policy trained largely on real-world data, without prior exposure to SceneSmith scenes, completed an apple-transfer task in simulation. Teleoperated robots also opened cabinets, stored bottles, and navigated rooms. Scene generation can take several hours, and the researchers hope added computing power and future 3D libraries will improve speed and support deformable objects.