AI agents build virtual training grounds for robots

Robots walking down the street surrounded by amazed passers-by are no longer a rarity. But these machines are still a long way from being universal assistants that could work in a kitchen or factory.

The main obstacle is data. Like humans, robots learn best through experience. But physically teaching them dozens of actions in different conditions is labor-intensive and takes a lot of time.

Simulation as a training ground

“It brings up the idea of using the simulation as a training ground. The physics engines that power robot simulators have advanced greatly in recent years, but one challenge remains: how to create enough rich and varied simulation content to capture the complexity of the real world,” says Russ Tedrake, Toyota Professor of Electrical and Computer Science, Aeronautics, Astronautics and Mechanical Engineering at MIT, and a principal investigator at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).

It turned out that AI agents - semi-autonomous programs that “think” and carry out clearly defined tasks - can help create those very realistic virtual worlds that robots need.

How SceneSmith works

The new SceneSmith system, developed by researchers at MIT CSAIL and the Toyota Research Institute, uses three agents to piece together objects, walls, and the overall appearance of a 3D scene. Its reconstructions of interior spaces - restaurants, bedrooms, hotels - are much more realistic and detailed than previous systems.

This allows robots to practice skills and try out different ways of performing tasks before they are turned on. And engineers save time on real tests.

How do agents get the idea of ​​what brick-and-mortar locations should look like? Each of them refers to a multimodal system - a visual language model (VLM), namely the advanced VLM GPT-5.2. It is trained on a huge amount of text and images from the Internet to work with visual cues.

This advanced model gives each agent a kind of spatial knowledge: first the “designer” agent generates the elements of the scene, then the “critic” evaluates how realistic it looks, and finally the “orchestrator” manages their dialogue, deciding when the scene is ready. Once the three VLMs complete co-creation, the scene can be directly loaded into the physics simulation software.

Creativity and diversity

“We found that the system was able to construct 3D scenes in a way that a human designer would,” says Nicholas Pfaff, an MIT EECS graduate student and CSAIL researcher and lead author of a paper published on the preprint server arXiv along with Tedrake. - We created over 1,300 scenes using the leading VLM with internet-scale a priori knowledge, and it produced incredibly creative and varied compositions. I didn’t put it into the system through hints - it just improvised.”

Thanks to VLM agents, you can ask SceneSmith to make, for example, "a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall" - and get a virtual playground full of objects for the robot to tinker with. Such rooms contain up to six times more objects per scene than previous methods.

This is great for helping robots learn skills like putting a cup in the sink, arranging fruit on plates, or moving a can of soda from a shelf to a table.

Checking readiness for real work

With so many rich virtual worlds at your fingertips, you can evaluate whether your robot is ready for real work, without unnecessary trial and error in the physical world. The researchers tested different action plans (also called “policies”) in SceneSmith digital worlds, creating one hundred unique spaces in the process.

The VLM agent evaluated each attempt - and it turned out that the robot's plans were flawed: the machine often failed its tasks. People agreed with the model's conclusions more than 99% of the time. This means that roboticists can weed out unsuccessful approaches in the simulation before the robot takes action in the real world.

But how realistic are these virtual worlds really? It is difficult to prove this directly, so researchers have approached the issue from different angles. The most revealing test: They took the robot's pre-trained policy—an AI controller trained mostly on real data and had never seen a SceneSmith scene—and ran it into generated environments.

In one test, users told the system, “Take an apple from a bowl and place it on a cutting board,” and the simulated robot did just that. If the scenes were not similar to the real conditions on which the policy was learned, nothing would have happened.

The team also controlled the robots remotely through virtual spaces, directing them to open cabinets, put away bottles and move between rooms. Experiments have shown that the media can withstand prolonged physical interaction and are not merely suitable for visual inspection.

Behind the Scenes: Three Agents

The agents used by SceneSmith play a clearly defined role in the generation process, creating scenes in stages. Essentially, they map out a room and bring it to life.

Let's say you want to create a scene that looks like the first floor of a house. The VLM “designer” starts with a general layout, the “critic” reviews it, and the “orchestrator” approves it. The agents repeat this approach at every step: adding furniture, placing objects on the walls, then on the ceiling, and finally placing objects that the robot can manipulate.

For example, VLMs could add cabinets that the robot can open and close—movable elements that were often missing from previous base models. At each stage, the second VLM makes sure that the scene is practical: say, it advises removing the bathtub from the living room.

The third VLM guarantees high quality of the scene, even rolling back the design process a few steps if the visuals fall short. Once the three VLMs complete a creative collaboration, real-world physics is added using simulation software.

Superiority over analogues

With a clear understanding of what rooms should look like, where to place objects, and how the laws of physics work, SceneSmith is noticeably superior to previous methods. Compared to basic scene generators like HSM and Holodeck, SceneSmith created environments with more objects - including a private office, a ceramics store, and even a Minecraft-style playroom.

SceneSmith has also become a favorite with over two hundred users. In more than 90% of cases, they found the system's visuals more realistic. In addition, they noticed that the system overall followed cues more accurately than other approaches. In other words, she was the best at generating those virtual platforms that users really wanted to see.

Strengths and Limitations

Realism, variety and richness of detail are SceneSmith's strengths. Even when it comes to creating individual 3D objects. You give the command to make a serving table on wheels - and the system generates a two-dimensional image, which it then turns into a detailed model with physical properties: mass, friction and inertia.

True, you have to pay for such detail in speed. It can take hours to create one scene - agents meticulously work out and check each object. Increase the computing power and the system efficiency will increase dramatically. CSAIL engineers also hope to expand the capabilities to deformable objects (like sponges) if extensive 3D libraries become available.

“SceneSmith is a major step forward. It offers an agent-based environment for generating simulation-ready interiors from a simple text query, says Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the study. “The system pushes the boundaries of what is possible: it increases the density of objects in the virtual environment, ensures their physical accuracy (rather than just visual plausibility) and creates assets that are not tied to a fixed library - they can be generated through text-to-3D.”

Publication data

Nicholas Pfaff et al., “SceneSmith: Agent-based generation of indoor simulation scenes,” arXiv (2026). DOI: 10.48550/arxiv.2602.09153

Journal information: arXiv

Who is behind this story? Lisa Locke. Bachelor of Arts, Master of Material Culture. Former museum editor, physician assistant and transplant coordinator. Editing for Science X since 2021. Full profile →

Andrew Zinin. Master in Physics with research experience. Longtime science news enthusiast. Plays a key role in the editorial success of Science X. Full profile →

This story is reprinted with permission from MIT News (web.mit.edu/newsoffice/), a popular site covering news about research, innovation, and teaching at MIT. Quote: “AI agents create virtual training grounds to help robots gain critical training data” (July 14, 2026). Retrieved July 15, 2026, from https://techxplore.com/news/2026-07-ai-agents-virtual-playgrounds-robots.html

This document is protected by copyright. Except for fair use for private study or research, no part of it may be reproduced without written permission. The content is provided for informational purposes only.