Wristband enables wearers to control a robotic hand with their own movements

MIT engineers developed an ultrasound wristband that tracks hand movements in real time and uses artificial intelligence to translate wrist images into positions of the five fingers and palm. The smartwatch-sized ultrasound sticker images muscles, tendons, and ligaments, while onboard electronics roughly cellphone-sized process the system. The approach is intended to capture the wrist structures that pull the fingers, rather than relying on external cameras, sensor gloves, or muscle electrical signals. Cameras can be obstructed, gloves can restrict natural movement, and electrical signals may be noisy or miss subtle motion changes.

The hand and thumb have 22 degrees of freedom. Researchers recorded volunteers’ gestures with surrounding cameras, matched the resulting hand positions to regions in ultrasound images, and labeled those regions. They trained an AI algorithm on the labeled images to associate image patterns with degrees of freedom, then found it predicted gestures from new images.

Eight volunteers with varying hand and wrist sizes tested the wristband. They made a range of gestures and grasps, including all 26 American Sign Language letters, and held a tennis ball, plastic bottle, scissors, and pencil. The system precisely tracked and predicted hand position in these tests.

In demonstrations, a wearer wirelessly controlled a commercial robotic hand in real time. The robot copied keyboard-playing motions to perform a simple piano tune and mimicked finger taps in a desktop basketball game. Paired with a computer program, pinching and grasping allowed users to zoom, move, and manipulate a virtual object continuously.

The team plans to reduce the hardware size and train the software on more gestures from people with broader hand sizes and shapes. They envision applications in virtual and augmented reality, video games, design tools, dexterous humanoid-robot training, and potentially surgical dexterity tasks.