An AI system for robots has been developed that predicts the consequences of actions

The robot has learned to think ahead.

A team of researchers led by Professor Jaeyoung Do from the AIDAS Laboratory in the Department of Electrical and Computer Engineering at Seoul National University developed Dynin-Robotics, a robotic foundation model that processes language, images, video and robot actions within a single system. The foundation model is a general-purpose AI trained on heterogeneous data to cope with a wide variety of tasks. The article was published on the arXiv preprint server.

Dynin-Robotics generates the robot's movements based on human instructions and camera images. The model can predict in the form of a picture what the scene will look like after a particular action or what the goal should be upon completion of the task, and uses these predictions to select and generate actions.

In tests on four real-life robot manipulation scenarios, Dynin-Robotics showed an average success rate of 78.4%, beating comparison models from NVIDIA and the American company Physical Intelligence involved in the experiments.

The article was accepted at NeurIPS 2026 - one of the leading international conferences on artificial intelligence and machine learning. The presentation is scheduled for December.

Instructions and forecasts come together

Robot hardware is evolving rapidly, but understanding new instructions and performing tasks in unfamiliar environments remains a major challenge. Unlike repetitive movements learned under given conditions, real-life tasks involve objects whose positions and shapes vary from situation to situation, and people's needs vary.

For example, to execute the command “Take the cup on the table, put it in the drawer and close it,” the robot must determine where the cup and drawer are located and build a sequence of actions: first move the cup, then push the drawer.

Recent research has explored using the language capabilities of large models like ChatGPT to control robots, as well as using video generative models to predict future scenes. To harness both the understanding of human instructions and the ability to predict the consequences of actions when generating movements, the team combined these abilities in one model.

The new technology expands on Dynin-Omni, an omnimodal AI model previously developed by the same group. Omnimodal artificial intelligence understands and generates multiple forms of information - language, images, video and sound - within a single model. By adding data about the robot's actions to this architecture, the researchers created a physical AI model that can perform real-world tasks like moving and stacking objects.

The key feature of Dynin-Robotics is that the model simultaneously learns to understand instructions and scenes, predict the results of actions and generate physical movements. It can also describe in sentences what exactly the robot did, based on a video of the task being performed.

Block stacking was assessed separately - in conditions without a given order and with the requirement to lay blocks in a certain color sequence.

Understanding the meaning and predicting the outcome

The researchers tested their unified training scheme on simulations of “picking fruit” and “arranging bouquets.” The model that was taught three things at once - to understand, predict and build actions - coped with unusually formulated commands better than the one that could only generate actions. It responded to both direct instructions like “put a rose” and indirect ones, where the object is named through the meaning: “put a flower that symbolizes love.”

Predicting a future scene helps the robot choose what to do next. A person, too, first imagines where he will put the object and what it will look like, and only then reaches for it. So is the robot: based on the current picture and command, it builds an image of what should happen after completing the task.

The team implemented two approaches - either the predicted scene serves as a guide for generating actions, or it is used to evaluate action options and select the appropriate one.

Acts faster - and in real conditions

At the same time, the researchers reduced the computation time required to generate actions. They group the information needed for a chain of actions and predict it in parallel. Data that is used repeatedly is also used again - this speeds up the development of the next action. The internal generation rate increased by approximately 29.2 times compared to the non-optimized version and was up to 36.6 times faster than Physical Intelligence's "π0.5".

In experiments with real robots, four manipulation scenarios were tested: moving fruit, sorting blocks by color, stacking blocks in a random order, and stacking in a given color sequence.

If we average across all four, NVIDIA “GR00T N1.6” had a success rate of 73.0%, NVIDIA “Cosmos Policy” had a success rate of 73.6%, and Physical Intelligence “π0.5” had a success rate of 76.9%. Dynin-Robotics showed 78.4% - the best average result among the models involved in the comparison.

Industry needs more testing

The technology will be useful where the robot has to take into account both human commands and the environment around it: say, when sorting and transporting goods in a warehouse or when working with parts in production. True, the current results were obtained in simulations and under predetermined conditions of real experiments. Before bringing the technology to the factory, it will have to be evaluated on a much larger variety of objects, tasks and operating conditions.

Building on Dynin's core series of models, the team plans to test the applicability of the design to parts processing, assembly, object sorting and other manufacturing and logistics applications. Another step is additional training for specific industries: the models will be adapted to the data and requirements of industry and medicine.

Robots that learn from experience

To achieve this, researchers are developing technologies for precise manipulation of robotic arms, recognition of contact states using tactile and force sensors, and rapid adaptation to new tasks and conditions. In parallel, joint work is underway with colleagues from the College of Medicine of Seoul National University: scientists are studying the use of AI in surgical assistant robots and in endoscopic robots for the gastrointestinal tract.

In addition, the team is working on AI agents that break down a human-set goal into several tasks, distribute them among robots, evaluate the results and adjust further actions. Well, let’s say the command “clear the table” comes. The agent breaks it down into separate steps, passes it on to the robot, checks what happened, and if something doesn’t work out, makes the robot try again, but in a different way.

Based on the accumulated experience - both successful attempts and failed ones - the researchers expect to create what they themselves call “a robot that becomes smarter with use.” Its ability to plan and execute tasks will continue to grow.

Do, who led the research, said: "What's important about this work is that one model simultaneously decides what the robot should do, how the environment around it will change after it moves, and what actions it should actually take." He added, “Building on this result, which extends Dynin-Omni's unified AI technology to robot control, we will continue to develop robotic intelligence that goes beyond laboratory experiments to understand human commands and cope with complex tasks in real-world environments.”

Hoeun Lee and colleagues. Dynin-Robotics: Omnimodal Unified Diffusion Vision-Language-Action Model. arXiv, 2026. DOI: 10.48550/arxiv.2609.13053

Journal: arXiv.

Citation

Robot AI predicts future scenes to guide its next moves (2026, October 19). Retrieved October 20, 2026, from https://techxplore.com/news/2026-10-robot-ai-future-scenes.html