Voice UI for The Blind

Mole-bile Voice is a framework for designing and developing usable voice interfaces for the blind. It is an ongoing project for my master’s thesis, in collaboration with another Berkeley student Pitan Hu.

Need Discovery

The project started as an opened exploration of using technology to solve a blind user’s challenge. We brainstormed many need areas and then conducted contextual inquiries with blind volunteers. 

We asked them how they exercised, cooked, used social media, and used fitness trackers, and observed them using a desktop computer, a smartphone, a dumbphone, an MP3 player, a braille typewriter, a refreshable braille display, a dollar bill value reader, and a microwave. At the end, we decided to focus on the following challenge:

Blind smartphone users have a hard time navigating their mobile apps.

For blind users, the navigation inside a smart phone app is achieved through voiceover, which provides a read-aloud of all elements on the screen. Here is an example of our user searching for flight information using the Orbitz iPhone app with voiceover. The user had much difficulty locating the functionalities she needed.

Proposed Solution

We hypothesized that a well-designed voice UI mobile apps would provide a better usability experience for blind users. To validate this hypothesis, we prototyped a Uber voice application (many blind people use Uber for transportation), and conducted within-subject usability testing between Uber voice and Uber GUI with voiceover (the existing way of interaction). Employing the extreme sampling technique, we tested with three blind users. The testing result was that all three users preferred the voice app over the GUI app, with two users strongly preferred and one user weakly preferred. 

Our Product

Our goal is to create a voice UI framework that defines usable design patterns and provides a SDK for easy development. Such a framework will encourage more voice UI application to be created, and thus benefit the blind population.

We evaluated the Siri and Amazon Echo, two primary voice services currently available:

VUI Feature

Question Answer

Procedural Task

Data Entry

Data Presentation



Limited to 1-2 interactions


No voice presentation



Limited to 1-2 interact'ns


Low Control & Freedom


Based on this evaluation, we decided that our framework will use the current technology for question answering and data entry, but improve the technology on procedural task and data presentation. 

Procedural Tasks

I went through four iterations and created this demo procedural flowchart for the Uber app using our Mole-bile Voice design framework:

We developed a functional prototype for the interaction flow and are currently conducting usability testing.

Data Presentation

I designed a mobile VUI for Obitz that explores interactions for long data entry and output.

Simple Procedural Tasks

To explore the interactions for simple procedural tasks, I designed a VUI NextBus app. 

In the next few months, we will continue user testing for the three demo apps, finish an interactive design console that helps voice UI designers create flowcharts like the above, and we will also develop an SDK that converts the flowchart into working code. Stay tuned as exciting things happen!


About the project name: We wanted a project name that would capture the idea of voice UI for blind users of mobile applications. We thought of mole, the cute subterranean mammal who moves nimbly underground. Moles have low-vision or no vision, as navigating underground doesn't require much sight. We want to spearhead a new way of navigation and the image of a mole fits our mission. That is how we came up with the project name Mole-bile Voice :)