[Update: Live Caption on “Select Phones”] Google works towards accessibility with Live Caption on Android Q, Live Relay, and Live Transcribe
We as consumers often take the world around us for granted. What we experience, we presume that the same is experienced by everyone around us in a similar sense, if not the same. This presumption goes on to extend to every part of our life, including technology. But disabilities are real, and living with them becomes a challenging task because of these presumptions. Accessibility thus becomes an important topic, and Google is doing its part in ensuring that people with disabilities have an equitable opportunity for enjoying these experiences. Android Q integrates several of Google’s accessibility efforts to make Android a much more cohesive experience, though not all of the features mentioned below are available in Android right now.
A lot of us never bother to take a second glance at caption settings, and we also consume a lot of media without even noticing the absence of captions. But for 466 Million people around the world who are deaf and hard of hearing, captions serve a purpose greater than convenience — they are the very medium of the experience. Android Q integrates Live Captions, allowing users from the Deaf community to access experiences with so much more ease and universality.
Once the setting has been enabled, Live Caption will automatically caption media that is playing audio on your device with a single tap. Live Caption works with videos, podcasts, audio messages, and any other app — even with stuff that is being recorded on the device. Captions will appear as soon as speech is detected playing on the device. And since all of it happens through on-device speech recognition, neither the audio nor the captions leave your phone, and you can use the feature without needing WiFi or cellular data.
Update 5/10/19: Google has confirmed with VentureBeat that Live Caption will be “coming to select phones running Android Q later this year.” Specifically, “select, higher-end devices,” according to Brian Kemler, Android accessibility product manager. The reason is apparently due to memory and space constraints. The initial rollout will be limited but will expand over time, and Google plans on releasing a list of devices that will get support for Live Caption as we near the first public Android Q release.
Furthermore, Google confirmed that saving transcription will not be possible with Live Captions (due to an intentional limitation in the AudioPlaybackCaptureConfiguration API), that it won’t work with phone calls, voice calls, or video calls (because it isn’t supported by the API), and that it’ll only support captioning in English at launch. Once the feature launches, an offline model will be downloaded, and updates to the model will be served through Google Play Services.
Live Relay builds upon the ideas put forth by Live Caption by allowing people to make and receive phone calls without having to speak or hear.
Live Relay uses on-device speech recognition and text-to-speech conversion to allow the phone to listen to an audio call and then speak out responses on behalf of the user who types the responses out. The feature works in tandem with predictive writing suggestion features such as Smart Compose and Smart Reply, making it easier to hold a live call by aiding quick responses. Live Relay runs entirely on-device, so calls are still private. Since Live Relay interacts with the other side through a regular phone call, it can also work with landlines on the other side.
While Live Relay would certainly be helpful for the Deaf community and Mute community, its use cases extend to situations wherein someone may not be able to speak or hear a phone call at that moment, but still wishes to interact with it. Google is also optimistic about integrating real-time translation capability within Live Relay, which in turn has the potential to let anyone call anyone else in the world and communicate regardless of language barriers.
Google states that Live Relay is still in the research phase. It is not immediately clear if the feature has been integrated into current Android Q builds — we are guessing that it will make its way to Android devices in the future.
Live Transcribe — Extension for users with speech impairments
Live Transcribe was shown off by Google earlier this year as a tool for deaf users to avail live transcription of speech around them. The app aimed to make everyday conversations more accessible by converting real-world speech through the phone’s microphone into real-time captions. Live Transcribe is already available as an early access limited beta through the Play Store, with support for over 70 languages and dialects. The app is also pre-installed on Pixel 3 devices.
Google’s latest efforts in improving accessibility extends Live Transcribe to not only deaf users but also for users who have speech impairments through Project Euphonia.
The team under Project Euphonia is using AI to improve the ability of a computer to understand diverse speech patterns, including impaired speech. Google has partnered with non-profit organizations such as ALS Therapy Development Institute and ALS Residence Initiative to record the voices of people affected by ALS, and then use these recordings to train AI models to more reliably transcribe words spoken by people with these kinds of speech difficulties. The current set of AI algorithms work with the English language to accommodate individuals who have impairments related to ALS, but Google is optimistic about the research being applied to larger groups and different speech impairments.
Further, Google is also building upon this by training personalized AI algorithms to detect sounds and gestures, and then take actions such as generating spoken commands to Google Home or sending text messages. This use case is particularly helpful to people who are severely disabled, cannot speak and can only interact with non-speech sounds and facial gestures.
These new functionalities do not appear to be live within Live Transcribe just yet. Google is requesting assistance from volunteers who have slurred or hard to understand speech, and if they are willing to record a set of phrases to help further train the system to work better. If you are eligible and wish to volunteer, please fill out the form for the same.
Google’s efforts on improving accessibility of technology is certainly commendable. We hope more software companies work towards providing differently-abled individuals an equitable experience of the world.