Robots acquire new skills from a single video in 29 seconds

Robots have learned to acquire new skills from a single human video in an average of 29 seconds. The new HOST framework acquires skills directly at inference time without disrupting what has already been learned.

The problem with conventional training

To work reliably in dynamic real-world environments, a robot must learn new skills quickly without lengthy additional training. Most current systems, however, perform well only on the tasks they were trained for.

Teaching a robot a new task takes time and money. Worse, additional training sometimes harms its performance on earlier tasks.

Researchers from Beijing Institute of Technology, X SQUARE ROBOT and Tsinghua University developed HOST (Human-to-robot One-Shot Skill AcquisiTion), a framework for faster, more efficient skill acquisition. Their approach, described in an arXiv paper, lets a robot learn a skill from one human video without sacrificing previously acquired abilities.

"The ability to acquire skills quickly and effortlessly while retaining those already mastered is essential for robots," write Guangyan Chen, Meiling Wang and their colleagues. "However, current methods still rely on a cumbersome training cycle: expensive, slow and corrosive to prior learning. We present HOST, a framework that lets a robot acquire skills in seconds from one human video without losing earlier skills."

How HOST turns video into actions

The framework developed by Chen, Wang and their team lets a robot learn a new skill simply by analyzing a human video. The process has three stages.

First, HOST uses camera frames to determine how far the robot has progressed through the procedure shown in the video. In other words, it identifies the part of the demonstration corresponding to the current moment.

The system then analyzes the next segment of the human demonstration and predicts what the robot should see as it progresses through the task. This intermediate step helps HOST account for differences between the demonstrator's body and the robot's physical structure.

Finally, HOST infers the actions the robot should perform from these predicted future observations. By continually identifying its progress, anticipating what it should observe next and selecting appropriate actions, the robot follows the demonstrated procedure while adapting it to its own body.

Although HOST acquires a new skill from one video without updating its parameters, the underlying architecture was pretrained on 193,462 robot trajectories covering 229 tasks. It was then adapted using 5,847 human demonstration videos, each accompanied by robot trajectories for the demonstrated tasks.

"HOST addresses skill acquisition through a cascade of self-anchored predictions. It first estimates the robot's progress within the demonstrated task, then translates upcoming progress into the robot's own future observations, and finally infers actions from those predicted observations," the authors write.

"This cascade is trained with video-anchored objectives: the robot trajectory and video are mapped into a shared task-progress space, and each objective is then redefined to match the video's future progress. HOST thereby lets the robot actively follow the demonstrated procedure and adapt it to its own embodiment."

Real-world tests and results

The researchers tested their system on a physical dual-arm robotic platform comprising two six-axis ARX R5 manipulators with parallel grippers and three RGB cameras.

The robot was tested on 50 previously unseen physical manipulation tasks involving different objects, tools and movements. It attempted each task 20 times, with objects placed in different starting positions or orientations. Human experts assessed the success of each attempt.

"HOST acquires new skills at inference time from a single human video in an average of 29 seconds and achieves a mean success rate of 62%," Chen, Wang and colleagues report. "This is 45% above the zero-shot baseline while preserving previously acquired skills. HOST even outperforms a base model fine-tuned on 50 robot demonstrations per task, requiring 50 times fewer demonstrations and acquiring each skill 507 times faster."

Initial tests show HOST's potential to teach robots new skills faster without extensive task-specific demonstrations. Future work could improve success rates and test the system on other robots or across a broader range of scenarios.