Boston Dynamics creates "Spot copilot" - like ChatGPT for a robot

Boston Dynamics is teaching its Spot robot to interact with the world around it using a large language model. The result is an interface similar to ChatGPT: in natural language, you can ask the robot to perform a task, and it will then figure out how to carry it out. This new direction, which the robot's creators call "Spot copilot", is an important step towards creating a truly universal robot.

“There is a factory with thousands of valves. Which one should you open?" asked a Boston Dynamics engineer. Spot, equipped with a large language model (LLM), understands the question and begins scanning the room. He finds a specific valve—the one under the most pressure—and turns it. The demo is part of the company's latest experiments in which mechanical "creatures" are coupled with AI to make them more useful and easier to control.

According to Boston Dynamics, the goal is to move from an approach in which a person gives precise instructions to a situation where a person specifies only the desired outcome. The company is shifting from low-level verbose commands to high-level natural language queries.

For example, instead of an operator manually guiding Spot with a joystick to the desired location, then pressing a button and triggering a carefully programmed script to flip a toggle switch, the operator could simply tell the robot, “Check the circuit breaker in the lab and turn it on.”

To make this possible, the Boston Dynamics team uses a ready-made LLM (which one is not specified) as a high-level planner. The operator's command is sent to the model. It then breaks down a complex task into a sequence of predefined skills or actions that the robot already knows how to perform.

These skills, which Boston Dynamics calls "Robot Atomic Actions" (RAAs), are the basic building blocks of the system. These include navigation (“Go to”), object recognition (“Find”) and object manipulation (“Pick”, “Place”, “Flip”). Think of LLMs as project managers and RAAs as individual workers, each with a highly specialized task.

The operator says, “Check the circuit breaker in the lab and turn it on.”

The LLM scheduler first decomposes the task into steps:

Find a laboratory.

Get to the laboratory.

Find the circuit breaker.

Reach the circuit breaker.

Turn on the circuit breaker.

The "find" and "navigate" commands are RAAs. But the phrase “turn on the circuit breaker” is still too abstract. Therefore, the LLM planner breaks it down further into specific physical actions:

Place the manipulator hand over the switch.

Lower the grip onto the switch.

Squeeze the grip.

Rotate your wrist 30 degrees.

The entire chain of reasoning is displayed to the user in the chat interface so that he can see how the robot interpreted the command. After this, the user can confirm the plan or ask to adjust it.

This approach, where the LLM acts only as a scheduler and produces code for predefined skills, is considered popular and practical in robotics today. It is generally considered safer and more reliable than giving direct low-level control of the robot's actuators to the LLM. In addition, the system is “tied” to the environment using visual systems: cameras help understand the world and objects in it.

For example, in the valve demonstration, the operator asked, “There are thousands of valves in the factory. Which one should you open?" The LLM logic was that the valve under the highest pressure is probably the most critical. Spot then used a thermal imaging camera to find the hottest valve (as a proxy for pressure) and a mechanical arm to turn it.

The system is imperfect. During a demonstration for IEEE Spectrum, Spot successfully found and switched a circuit breaker, although the first attempt failed because the robot's arm was not high enough to clear the barrier. The robot recognized the error, diagnosed the cause and adjusted the plan, raising its hand higher on the second attempt. This demonstrates the level of autonomous error recovery.

Boston Dynamics is taking the same LLM planner approach to its humanoid robot Atlas and the new electric version of Spot. The company sees this as a key part of its strategy to create robots that can work in a variety of real-world environments, from factories and construction sites to power plants and homes, without the need for time-consuming specialized programming for each site. The ultimate goal is a robot that can understand and follow commands as easily as asking a human for help.