Human Activity Data Collection System Presented for Robot Training

A new portable system, EgoExplorer, recorded 100 hours of human activity to train humanoid robots. The dataset combines first-person video, body movements, finger articulation and tactile data.

Why do robots need human examples?

Humanoids - robots with human-like bodies - could take on a variety of housework and routine manual tasks. But to do this, they need to reliably coordinate their movements, posture and the way they touch or grasp objects.

Human examples of task performance provide valuable clues for humanoids. True, collecting data on people that will really help in training robotic algorithms can be very difficult.

How the capture system works

Researchers from Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University have developed a wearable system. It captures data while people perform different actions.

The system combines a GoPro HERO13 head-mounted camera with five VIVE Ultimate Trackers, MANUS Metagloves or Wuji gloves with sensors, and custom 3D printed mounts and straps.

The head camera records video from a first-person perspective. Five trackers - one on the head, one on the back of each hand and one on each ankle - track their position and orientation.

Sensor-equipped gloves record the wearer's finger movements, and Wuji also captures tactile data. Fasteners and straps simply securely fix the equipment.

Dataset EgoExplorer

Using this wearable system, the researchers recorded 100 hours of data while people performed a variety of activities. The collected data was combined into a new dataset called EgoExplorer.

“With a focus on task performance, continuous recordings combine egocentric video with reconstructed full-body movement, fine-grained finger articulation, and tactile observations accompanied by step-level descriptions,” the authors wrote.

Pipeline Calibrate-Complete-Constrain

Zhou, Jiang and colleagues presented a motion reconstruction pipeline. It is a sequence of processing steps that turns data recorded by a wearable system into full-body motion estimates.

“To achieve such coverage with sparse body sensing, we developed the Calibrate-Complete-Constrain pipeline,” the researchers wrote. “Specifically, reusable calibration captures the geometry of the participant and sensor mounts prior to motion output to decouple mount displacements from articulation.”

The developed software estimates how the joints moved when sensors were not directly tracking them. It does this by using previously learned movement patterns.

The program then checks that the reconstructed movements match the sensor data, remain smooth over time, and match the shape of the person's body.

From demonstrations to simulations

The team compared their motion reconstruction method to earlier approaches. Errors in body positioning relative to reference trackers have decreased.

Preliminary experiments yielded an interesting result: joint generation of video, motion and haptic data works better than generating each stream separately.

The researchers also trained a controller for a simulated humanoid robot, Unitree G1. They used two recordings of movements from the collected database of human demonstrations as a basis. The simulated robot completed both sequences without falling.

However, the experiments did not establish precise object interaction or good performance on unfamiliar tasks. And the method was not tested on physical robots.

“Relative to tracker benchmarks, the method reduces body point error by 71.9% compared to the strongest baseline,” the authors wrote. “Matched scores also show improvements in hand stability and contact scores.”

Application prospects

In the future, the EgoExplorer dataset can be used to train other robotic controllers in simulation. Ultimately, controllers trained on these recordings could help humanoid robots better perform tasks in a variety of real-world environments.