Penn State researchers have developed a new navigation tool that could reshape digital assistance for people with visual impairments.<\/p>\n
The smartphone-based system, called NaviSense, uses artificial intelligence to identify objects in real time and guide users toward them with audio and haptic cues.<\/p>\n
The team unveiled the technology at the ACM SIGACCESS ASSETS \u201925 conference in Denver, where it won the Best Audience Choice Poster Award.<\/p>\n
NaviSense aims to fix long-standing problems in assistive navigation software. Many current tools depend on human support teams. Some rely on object libraries that must be preloaded in advance.<\/p>\n
Vijaykrishnan Narayanan, Evan Pugh University Professor and A. Robert Noll Chair Professor of Electrical Engineering, said this limits flexibility.<\/p>\n
\u201cPreviously, models of objects needed to be preloaded into the service\u2019s memory to be recognized,\u201d Narayanan said.<\/p>\n
\u201cThis is highly inefficient and gives users much less flexibility when using these tools.\u201d He said the team turned to AI to break this bottleneck.<\/p>\n
The app connects to an external server running large-language models and vision-language models. <\/p>\n
These systems allow NaviSense to interpret voice prompts, scan the surroundings, and identify targets without relying on static object databases.<\/p>\n
\u201cUsing VLMs and LLMs, NaviSense can recognize objects in its environment in real-time based on voice commands, without needing to preload models of objects,\u201d Narayanan said. \u201cThis is a major milestone for this technology.\u201d<\/p>\n
Built with user input<\/p>\n
The team shaped the app after extensive interviews with visually impaired<\/a> participants.<\/p>\n Ajay Narayanan Sridhar, a computer engineering doctoral student and lead student investigator, said these sessions helped map out real-world needs.<\/p>\n \u201cThese interviews gave us a good sense of the actual challenges visually impaired people face,\u201d Sridhar said.<\/p>\n NaviSense<\/a> listens for a user\u2019s spoken request, searches the space, and filters out irrelevant objects. When the system needs clarification, it asks follow-up questions.<\/p>\n The conversational feedback offers flexibility that many existing tools struggle to provide.<\/p>\n One of the app\u2019s standout features is hand guidance. The system tracks the user\u2019s hand by monitoring the phone\u2019s movement and then gives directional cues that help them reach the object.<\/p>\n Sridhar said this capability filled a major gap. \u201cThere was really no off-the-shelf solution that actively guided users\u2019 hands to objects, but this feature was continually requested in our survey,\u201d he said.<\/p>\n Strong early performance<\/p>\n The team tested NaviSense with 12 participants in a controlled environment. Users compared it with two commercial options. <\/p>\n Researchers measured how quickly each tool identified objects and how accurately it guided users.<\/p>\n NaviSense reduced search time and provided more precise detection. Participants also reported a better overall experience.<\/p>\n One user wrote, \u201cI like the fact that it is giving you cues to the location<\/a> of where the object is, whether it is left or right, up or down, and then bullseye, boom, you got it.\u201d<\/p>\n The team is now refining power consumption and improving model efficiency. Narayanan said the tool is nearing commercial readiness. <\/p>\n \u201cThis technology is quite close to commercial release, and we\u2019re working to make it even more accessible,\u201d he said.<\/p>\n