Penn State researchers have developed a new navigation tool that could reshape digital assistance for people with visual impairments.
The smartphone-based system, called NaviSense, uses artificial intelligence to identify objects in real time and guide users toward them with audio and haptic cues.
The team unveiled the technology at the ACM SIGACCESS ASSETS ’25 conference in Denver, where it won the Best Audience Choice Poster Award.
NaviSense aims to fix long-standing problems in assistive navigation software. Many current tools depend on human support teams. Some rely on object libraries that must be preloaded in advance.
Vijaykrishnan Narayanan, Evan Pugh University Professor and A. Robert Noll Chair Professor of Electrical Engineering, said this limits flexibility.
“Previously, models of objects needed to be preloaded into the service’s memory to be recognized,” Narayanan said.
“This is highly inefficient and gives users much less flexibility when using these tools.” He said the team turned to AI to break this bottleneck.
The app connects to an external server running large-language models and vision-language models.
These systems allow NaviSense to interpret voice prompts, scan the surroundings, and identify targets without relying on static object databases.
“Using VLMs and LLMs, NaviSense can recognize objects in its environment in real-time based on voice commands, without needing to preload models of objects,” Narayanan said. “This is a major milestone for this technology.”
Built with user input
The team shaped the app after extensive interviews with visually impaired participants.
Ajay Narayanan Sridhar, a computer engineering doctoral student and lead student investigator, said these sessions helped map out real-world needs.
“These interviews gave us a good sense of the actual challenges visually impaired people face,” Sridhar said.
NaviSense listens for a user’s spoken request, searches the space, and filters out irrelevant objects. When the system needs clarification, it asks follow-up questions.
The conversational feedback offers flexibility that many existing tools struggle to provide.
One of the app’s standout features is hand guidance. The system tracks the user’s hand by monitoring the phone’s movement and then gives directional cues that help them reach the object.
Sridhar said this capability filled a major gap. “There was really no off-the-shelf solution that actively guided users’ hands to objects, but this feature was continually requested in our survey,” he said.
Strong early performance
The team tested NaviSense with 12 participants in a controlled environment. Users compared it with two commercial options.
Researchers measured how quickly each tool identified objects and how accurately it guided users.
NaviSense reduced search time and provided more precise detection. Participants also reported a better overall experience.
One user wrote, “I like the fact that it is giving you cues to the location of where the object is, whether it is left or right, up or down, and then bullseye, boom, you got it.”
The team is now refining power consumption and improving model efficiency. Narayanan said the tool is nearing commercial readiness.
“This technology is quite close to commercial release, and we’re working to make it even more accessible,” he said.
The project received support from the U.S. National Science Foundation.