Cornell researchers have developed a system that uses a smartwatch’s speaker and microphone to track movement throughout a person’s body, potentially opening a simpler route to exercise monitoring and physical rehabilitation.
Called EchoMotion, the system sends out ultrasonic sound and uses a small artificial intelligence model to interpret echoes from moving limbs. In tests with an unmodified Samsung Galaxy Watch 6, it estimated joint locations with an average error of less than four inches for people new to the system, according to Cornell’s Nov. 11 report.
Using echoes to follow movement
The work comes from Cornell’s Smart Computer Interfaces for Future Interactions, or SciFi, Lab, directed by information science associate professor Cheng Zhang. Lead author Saif Mahmud, a doctoral student in information science, said conventional wearable sensors generally measure movement of the device itself. Tracking the rest of the body has often required additional equipment, such as phones, earbuds or virtual reality headsets.
EchoMotion instead interprets sound reflected from the wearer’s body. That approach avoids recording camera images of the wearer and nearby people, while producing less data to process, according to the researchers.
“This is a very important jump for body tracking,” Mahmud said in Cornell’s report.
The researchers first built a 3D-printed watch replica with two speaker-and-microphone pairs. Its speakers emitted inaudible ultrasonic signals, and its microphones recorded echoes returning from moving limbs.
Ten volunteers wore the device on their nondominant hand while performing 18 exercises. The team used those recordings to develop a deep learning model, checking its estimates for 12 joints, from the head to the ankles, against body positions recorded by cameras.
Testing commercial watches
Commercial smartwatches have a single speaker-and-microphone pair, so the team next tested EchoMotion on an unmodified Galaxy Watch 6 with 15 new volunteers. A one-time, 12-minute calibration reduced the average joint-location error by more than a third, to about 2.4 inches.
The same AI model also worked on a Samsung Galaxy Watch 7 and Google Pixel Watch 3 without retraining, Cornell reported.
Power consumption was another focus. Prototype sensing used about 12 milliwatts, roughly 40 times less than a wearable camera, according to the university. A 30-minute session on the Galaxy Watch 6 consumed 5% to 6% of its battery.
“We have made significant efforts in making EchoMotion work on commercial devices, so that the adoption can be very easy,” Mahmud said.
For testing outside the lab, 10 additional participants performed the exercises in a small room, a narrow hallway and beside a noisy road. Furniture and ambient noise did not interfere with performance in those tests, Cornell said. Accuracy suffered slightly in the hallway because of echoes from other people.
Mahmud expects training with data from many more users could help the model interpret movement in crowded settings. That remains an anticipated improvement rather than a demonstrated result from the reported tests.
Fitness tools remain in development
The team is developing fitness applications that could count repetitions and help users adjust exercise form. Researchers are also exploring combining EchoMotion with cameras on smartglasses to track objects a person handles. Workout monitoring, daily activity tracking and rehabilitation are potential uses; the report describes research and development rather than announcing a consumer release.
The research paper was presented Oct. 13 at a computing and wearable technology conference in Shanghai. Other co-authors include François Guimbretière, Jiwan Kim, Tianhong Catherine Yu, Kenny Liang and Vineet Parikh. The National Science Foundation supported the work.
FingerLakes1.com is the region’s leading all-digital news publication. The company was founded in 1998 and has been keeping residents informed for more than two decades. Have a lead? Send it to news@fingerlakes1.com.