The tool will come to other devices in the future.
Smartphones have long offered voice-to-text transcription, but the same can’t be said for sign-language-to-text. That’s changing today with the launch of the Pixel 11 family, which marks the debut of DeepMind’s new sign-language-to-text (SL2T) model. Google has embedded the model inside of Gboard and Live Transcribe, where it will allow deaf and hard of hearing users to sign to their phone anywhere where they would have normally typed in the past. In practice, that will give those users a faster, more natural way to search the web, write messages and converse with Google’s Gemini chatbot.
“There is great diversity among deaf people in terms of their level of proficiency in signing, speaking, reading, and writing, so it is important to support access in all modalities,” says DeepMind. “Deaf people can benefit from sign language processing in the same way that hearing people benefit from spoken language processing, plus the technology opens new possibilities for bridging the communication gap between Deaf and hearing communities.”
DeepMind trained SL2T on more than 100,000 hours of multilingual sign language data, with approximately a quarter of the dataset representing information in American Sign Language (ASL). For that reason, the model only supports ASL to English transcription at launch. Google plans to support more languages and devices in the future.
SL2T doesn’t interpret raw video. Instead, a separate on-device model converts footage into a sort of wireframe of geometric coordinates that are sent to Google’s servers. The company says the model is designed in this way to protect user privacy. SL2T also differs from past sign-language-to-text systems in that it translates the wireframes it receives directly into text, instead of producing intermediate known as glosses.
“Glosses fail to capture rich, non-linear aspects of sign languages such as non-manual markers and spatial constructions,” DeepMind explains. “Translating directly from landmarks removes artificial vocabulary limits and allows translation quality to scale directly with data.”
While support for ASL is a great start, DeepMind acknowledges there’s more work to be done. Globally, there are more than 70 million deaf and hard of hearing people who communicate in some 200 sign languages. The good news is the company has a hard start there since SL2T was simultaneously trained on multiple languages so that it could learn the shared underlying structures between.
