Researchers from Cornell University and KAIST have developed a system that turns ordinary smartwatches into hand-tracking devices using AI-powered sonar. <\/p>\n
The technology, called WatchHand, uses the built-in speaker and microphone in off-the-shelf smartwatches to track finger and wrist movements in real time.<\/p>\n
The system works by emitting inaudible sound waves from the smartwatch. These waves bounce off the user\u2019s hand and return to the microphone, creating an echo profile.<\/p>\n
A machine learning algorithm processes this data directly on the device to estimate hand pose in three dimensions.<\/p>\n
Unlike existing wearable hand-tracking systems that rely on cameras or bulky external sensors, WatchHand requires no additional hardware. This makes it significantly more practical for everyday use and scalable across millions of existing devices.<\/p>\n
The researchers say the goal is to make the human hand itself an input device for interacting with computers and other digital systems, reducing reliance on keyboards, mice, and touchscreens.<\/p>\n
Turning hands into input<\/p>\n
\u201cIn the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch,\u201d said Chi-Jung Lee, a doctoral student at Cornell and co-lead author of the study. \u201cOur hands can act as an input device with computers.\u201d<\/p>\n
The system was tested on 40 participants across four separate studies, collecting around 36 hours of gesture data. It was evaluated across multiple smartwatch models, different hand orientations, and noisy environments. <\/p>\n
The results showed that WatchHand could reliably track finger movements and wrist rotations under varied conditions.<\/p>\n
The technology could enable a range of applications, including gesture-based control for computers, augmented and virtual reality<\/a> systems, and assistive tools for users with limited mobility or speech.<\/p>\n \u201cWatchHand substantially lowers the barriers to hand-pose tracking,\u201d said Jiwan Kim, a doctoral student at KAIST and co-lead author. \u201cIf any device has a single speaker and microphone, our approach is applicable.\u201d<\/p>\n Software unlocks new sensing<\/p>\n The system processes all data locally on the smartwatch, addressing privacy concerns associated with cloud-based tracking systems. <\/p>\n This also reduces latency, enabling real-time interaction without requiring external computation.<\/p>\n However, the technology still has limitations. It currently works only on Android-based smartwatches<\/a> and struggles to maintain accuracy when the user is in motion, such as walking. Researchers are working to improve performance under these conditions.<\/p>\n \u201cWatchHand reflects my lab\u2019s broader vision of transforming everyday wearables into intelligent behavior-sensing platforms,\u201d said Cheng Zhang, associate professor at Cornell.<\/p>\n