TOKYO – July 27, 2021 – Preferred Networks, Inc. (PFN) has developed a deep learning-based body and hand pose estimation technology that recognizes precise human body movements in camera images including finger positions and overlapped hands. PFN is currently seeking collaborative partnerships to apply the new technology.

Upper left: PFN’s 3D pose estimation technology recognizes the hand gesture meaning “to match” in Signed Japanese.
Upper middle and right: Sign language avatars for SoftBank’s SureTalk system automatically generated with the recognized gesture
Lower: Recognized results plotted on 3D axis 

The new pose estimation model has been trained on PFN’s supercomputer with computer-simulated 3D human body images to achieve high recognition accuracy. The new technology can recognize body and hand features in a video recorded with a regular camera or a smartphone, instead of requiring a specialized camera or body-mounted sensors. PFN’s pose estimation technology has drastically reduced false detections of fingertip positions and errors due to overlapped hands, which were common issues in previous technologies. PFN expects the improvement will enable companies to digitize precise hand movements in music, sports and crafts, and the technology can be adopted to telediagnosis, telerehabilitation as well as automated avatar production.

The new 3D pose estimation technology has been adopted for SureTalk, the mutual sign language-to-voice communication system developed by SoftBank for users in Japan. The system used PFN’s technology to track precise movements of fingers, wrists, elbows, shoulders, neck, nose and hip in sign language users’ videos collected by SoftBank to automatically generate computer graphics avatars using standardized Signed Japanese. The system required precise 3D pose estimation since overlapped hands and finger positions affect the meanings of sign language. The automatically generated avatars were reviewed by multiple sign language users to ensure accuracy and clarity.