Spelling out the letters A, S, and L to represent American Sign Language, image courtesy of Pixabay.
While there is “talk to text,” there’s no equivalent tool for American Sign Language (ASL) to be automatically recognized and translated into text. New research and language technologies developed by scholars affiliated with the USC School of Advanced Computing’s Thomas Lord Department of Computer Science might help future researchers who aim to build translation tools.
The team’s innovations outlined in paper presented at the 2025 Nations of the Americas Chapter of the Association for Computational Linguistics conference, are in developing a machine learning model that treats sign language data as a complex linguistic system rather than just a mere translation of English. The team led by Lee Kezar, then a doctoral candidate in computer science out of Professor Jesse Thomason’s GLAMOR (Grounding Language in Actions, Multimodal Observations, and Robotics) Lab, introduces a new natural language processing model, incorporating the spatial and semantic richness of ASL, treating it as a primary language with its own syntax.
The first step for developing a means of ASL recognition which demands an understanding of the language’s specific nuances—and how natural signing may be divided into phonological features, such as the ‘C handshape’ or ‘produced on the forearm’” for a computer.
(Shown here by Dr. William Vicars and Lifeprint.com)
However, the main challenge for creating a model for automatic ASL detection and other global sign languages—is the limited data available. In contrast, says corresponding author Thomason, data for non-signed languages is available from all over the world via the internet and films.
Thus, the team realized that one of the first steps needed was to generate a knowledge graph, an organized way of communicating graphically how the visual properties of signs relate to their meanings throughout the lexicon.
(For example, the C handshape in CUP below shows the shape of the cup itself.)
A GIF of the word “cup” in American Sign Language
Kezar, who knows American Sign Language and is now a Postdoctoral Researcher at Gallaudet University, took on this project as he saw a huge gap in this family of languages.
“Sign languages are full, natural languages. They’re complete, meaning we can express basically any idea. But it’s not really included in natural language processing research,” said Kezar.
The researchers, who were sure to include native signers and collaborated with the Deaf community, explain that any viable model for ASL recognition and generation would need to take in account some unique aspects of the language that make up the signs, including:
Facial expressions as noted here ins the signs for “understand” versus “don’t understand.”
Understand
Don’t understand
Where a sign is in relation to a body as seen in the signs summer versus dry.
Summer
GIF of the word “Summer” in American Sign Language
Dry
In addition, any model needs to have flexibility to recognize the new signs that are evolving.
For example, the sign for the coronavirus:
Thomason emphasized that the project is not simply about recognition, but also about understanding and generation—creating systems that can comprehend and produce fluent sign language in its natural structure.
Thus far, the researchers have trained a machine learning model to achieve:
91 percent accuracy in recognizing isolated signs.
14 percent accuracy in recognizing unseen signs’ semantic features, such as inferring that a sign is related to sight because it involves the V handshape produced near the eyes (e.g. SEE, LOOK, REVIEW).
(See)
They have also trained a machine learning model to achieve 36 percent accuracy at classifying the topic (news, sports, etc.) of ASL videos on YouTube.
To achieve this, the researchers emphasized the importance of working directly with the Deaf and Hard-of-Hearing community, including native signers and linguistic experts, to guide the direction of the models and how data is handled.
“We wanted to do something that is deeply respectful of the language itself,” he noted. “And that meant collaborating directly with members of the Deaf community,” said Thomason.
As the project moves forward, the team will look to expand their model to include other sign languages around the world by mapping out their shared grammatical structures and unique features.
Ultimately, Kezar says the team “envisions applications ranging from looking beyond automatic translation, and focusing on more useful applications allowing users to search YouTube using ASL (e.g. the topic modeling experiment), building augmented-reality educational tools (the focus of Kezar’s postdoc), and enabling linguistic research into signing.”
Published on September 8th, 2025
Last updated on September 8th, 2025