A Harvard-developed smartphone app helped blind and low-vision users walk an outdoor route 13 percent faster than their usual navigation setup. It also cut obstacle collisions indoors by roughly 41 percent. The app, called Mobilio, runs entirely on a standard smartphone and requires no extra hardware beyond what most people already carry in their pocket.



Engineers at Harvard’s John A. Paulson School of Engineering and Applied Sciences built the app, combining machine learning, sensor fusion, and personalized audio cues into turn-by-turn guidance. Ph.D. student Raymond Liu led the research alongside Patrick Slade, an assistant professor of bioengineering.



What users need



Before writing any code, the team surveyed more than 100 blind or low-vision people about their navigation needs. Three capabilities came up repeatedly: reliable turn-by-turn directions, continuous guidance along sidewalks and paths, and real-time obstacle detection.



Existing tools rarely cover all three at once, according to the researchers. Canes and guide dogs help with immediate surroundings but offer no route guidance. Standard GPS navigation apps handle turn-by-turn directions but lack the precision blind or low-vision users need to navigate safely.



A pedestrian’s view



Mobilio pulls data from a smartphone’s camera, GPS, motion sensors, and LiDAR sensor where available. At its core sits a custom computer-vision model that scans the live camera feed to identify walkable paths from a pedestrian’s perspective.



Slade described the system as functioning like a small autonomous vehicle’s navigation plan, built around the phone’s sensors, GPS data, and the user’s actual movement. Liu said most publicly available street-scene datasets come from cars rather than pedestrians, which causes standard computer-vision models to misclassify sidewalks and crosswalks. He trained a segmentation model specifically on pedestrian-view images to correct that gap. He also worked to ensure every algorithm could run in real time on ordinary smartphone hardware rather than specialized computing equipment.



Beeps that adapt



Rather than relying only on spoken directions, Mobilio delivers continuous directional beeps as a person walks. The beeps indicate whether to steer left or right. The system uses human-in-the-loop optimization to track how accurately each user follows the cues, then automatically adjusts pitch and timing patterns to fit that individual.



Researchers tested the app with 14 volunteers recruited from the Carroll Center for the Blind in Newton, Massachusetts. Each participant completed both an outdoor route and an indoor obstacle course, first using Google Maps navigation with a white cane, then again using Mobilio with a cane. Beyond the speed and collision improvements, Mobilio’s overall reliability in guiding users to their destination matched that of a human guide.



Liu said the project carries personal significance, since his older brother is blind and helped shape the work’s goals around real independent navigation. He emphasized that testing with actual users was essential throughout development, given how varied blindness and visual impairment can be from person to person.



The team next plans to test Mobilio across a wider range of real-world settings around Boston. They are working toward a broader public release with support from the Harvard Grid Accelerator.



The findings were published in the journal Nature Biomedical Engineering.

Harvard builds a smartphone navigation app for blind users as reliable as a human guide