Can AI tell you where you left your keys?

Researchers at the Massachusetts Institute of Technology (MIT) have created a long-term memory system for robots. It allows machines to quickly form and reproduce detailed mental models of complex spaces.

The new method combines advanced cartographic representations with detailed descriptions of the environment. The robot collects this data during long movements and instantly accesses memory to answer complex questions in ordinary language.

The problem of space-time memory

A car factory worker can remember in which container she left an unassembled part the night before. She quickly returns to the same place to pick it up. But the robots who work side by side with her have a hard time with such “spatio-temporal” memory.

An article about the development was published on the arXiv preprint server. In the future, this technology will allow a worker to simply ask a robot assistant: “Go and bring the part we started assembling last night,” and it will complete the task.

The memory system responds to requests more accurately than modern analogues. It works fast enough for real-time use by a mobile robot.

In addition to robotics, the method can find application in augmented reality systems. For example, helping repairmen identify faults or give directions to passengers.

“If we want robots to work alongside people and interact better with them, they need to speak the same language. A robot must be able to reason about time and space in the same way as a human. Essentially, this is what our method does. It turns a regular map into a language-based map that is easier for a robot to manipulate and access using language,” explains Luca Carlone, assistant professor of aeronautics and astronautics at MIT (AeroAstro), principal investigator of the Laboratory for Information and Systems Solutions (LIDS), and director of the SPARK Lab.

The paper is co-authored by lead author Nicholas Gorlo, an MIT graduate student, and Lukas Schmid, a former MIT research fellow and now a professor at the Nuremberg University of Technology in Germany. The research was recently presented at the Conference on Computer Vision and Pattern Recognition (CVPR).

How it works

Memory allows artificial intelligence systems - the same chatbots - to answer complex questions and analyze previous dialogues with the user.

“We want to create a new type of memory - spatio-temporal. Such that the AI-based robot remembers real interactions and data from sensors. Like ChatGPT, but tied to the physical world and able to answer any question about the environment. Like, 'Where did I leave my wallet?'" Carlone explains.

To develop such a memory system, MIT researchers combined two fields: computer vision and robotic mapping.

Multimodal computer vision models can recognize and describe in detail objects in a scene. But, as a rule, only one annotation is processed at a time. And frameworks for robotic mapping build 3D maps of the environment - an entire apartment or a university campus. But they often lack detailed descriptions of objects or require enormous computing resources.

The method, which was invented at MIT, is called Describe Anything, Anywhere, Anytime, at Any Moment (DAAAM). It takes the best of both worlds.

Using DAAAM, the robot, moving through the environment, attaches detailed descriptions to the objects it sees. Let's say it notes that a particular building on the MIT campus is called the Stata Center and is designed in a certain architectural style. Or that there are five bicycles in the bike rack, and the red one has a flat tire.

All this detailed information is stored in a 3D map, organized according to a spatial principle: objects are grouped into separate zones. This is how the robot remembers that a red bicycle with a flat tire is parked in the Stata Center parking lot.

But existing technologies that can collect such rich descriptions spend several seconds annotating a couple of objects. This is too slow for real-time work - after all, in a few minutes of exploration, the robot can see hundreds of objects.

“The faster a robot develops spatial memory, the more effective it is in its environment,” Carlone adds.

Speeding up the process

To speed things up, DAAAM groups nearby objects as you move. Using the optimization method, the system selects key frames for annotation - images with the clearest overview of several objects at once. It describes them in parallel, which speeds up calculations by ten times.

As the robot explores space, it links each batch of annotations to multiple objects at a specific point on the 3D map.

“We annotate each object only once, so our framework can run in very large environments in real time. And by grouping objects into zones, he is able to respond to a variety of requests - both about objects and about places,” explains Gorlo.

Once spatial memory is formed, the system needs to efficiently retrieve information from a huge database of objects and descriptions. To do this, the researchers used LLM, which calls up various tools and quickly finds the necessary data, reducing the risk of hallucinations. Thanks to this, DAAAM responds to the user's request accurately - in just a few seconds.

For example, if you ask a robot about a sculpture it saw near one of the buildings on the MIT campus, DAAAM can use semantic search for the word “sculpture” or another tool to search by the location of the building.

In tests, compared to other methods, DAAAM showed 21-53% higher accuracy - depending on the type of question.

In the future, researchers want to extend DAAAM so that the system can capture significant events occurring in the environment. They are also working on implementing confidence levels into the system's responses.

“Ultimately, we want robots to help with any task. With this framework, we are laying the foundations for a universal agent that can do anything you ask,” concludes Gorlo.