Apart from having a great camera, Google's Pixel lineup also offer many useful software features. Take 'Hold For Me' as an example: This feature lets the Google Assistant take your place on a phone call while you're waiting on hold for a business to get back to you. When Google makes a new feature that leverages its existing services, the result can be phenomenal. For the upcoming Pixel 6 series, Google plans to do just that by integrating parts of Google Lens, Translate, Assistant, and Live Caption into one feature called Live Translate. Thanks to our source who has access to the unreleased Pixel 6 Pro, XDA can offer an exclusive first look at Live Translate through some screenshots of the feature.
Last week, a source with access to the Pixel 6 Pro reached out to XDA, allowing us to share some new details on the phone's hardware. After speaking with our source, we discovered that their Pixel 6 Pro features an unreleased build of the Android System Intelligence app, which was previously called Device Personalization Services. This app is responsible for providing several software features on Pixel phones, including Live Caption, Screen Attention, and more. As we previously discovered, this app will also contain the service for the new Live Translate feature, which will debut on the Pixel 6 series but may also come to older Pixel phones running Android 12.
Live Translate was one of the only new software features that Google confirmed when it kicked off its Pixel 6 marketing blitz last month. At the time, Google didn't explicitly confirm the name of the new feature, but they did tell publications like The Verge and The Washington Post that the Pixel 6 will be able to generate live, translated captions of content you're watching or listening to. The Verge was even able to see a demo in-person of the feature transcribing and then translating French to English in real time, but they sadly weren't allowed to share any images or videos from the demo. We managed to get the Live Translate feature to show up on our own device, though, so we're able to share some screenshots.
Setting up Live Translate
Starting off with the setup flow for Live Translate, we can already see that there's an aspect to this feature that went previously unreported. Live Translate will not only let you translate captions into the language of your choice, but it'll also translate messages, text detected in the camera viewfinder, and act as an interpreter, depending on the language you choose.
After completing the initial setup process, Live Translate can be found under Settings > System. Here, you'll be able to toggle the feature, choose whether you want to allow downloading language models over mobile data, choose the language you want to translate to, and download new language models. The language models range in size from 50-200MB depending on the supported features. Choosing Japanese, for example, will let you translate messages, captions, and text in the camera viewfinder, while choosing Mandarin will only let you translate messages and text in the camera.
Here is the full list of supported languages in Live Translate and what features are supported for each language:
We weren't able to get any of the below Live Translate features working on our Pixel 3 XL, but our source told us it works just fine on their Pixel 6 Pro, as is expected. According to Google, Live Translate works fully offline and is accelerated by the new Google Tensor chip in the Pixel 6 series, so it's possible we're missing a few things needed to get it working on Google's Snapdragon-powered Pixel phones. Thus, we're unable to share screenshots of the feature actually working at the moment, but we do have more screenshots and information that tell us a lot about the feature.
Translating captions
In order for Live Translate to translate captions, you'll first need to turn on the Live Caption feature. This is currently accessible under Settings > Sound & vibration on Android 12, but it seems that Google will be adding a new Settings entry titled "Languages & translation" where you'll be able to choose which languages you want captioned and translated to. Coupled with the above list, this new settings page confirms that Live Caption will soon support additional languages beyond English. These new languages include French, German, Italian, Japanese, Portuguese, and Spanish. Live Caption on Chrome OS already supports these languages (save for Portuguese), so we're not surprised to expanded language support coming to Android.
Translating messages
We don't know exactly how Live Translate will handle message translations, but a quick analysis of the new Android System Intelligence app reveals a few illuminating details. It sounds like the feature will automatically detect when a message you receive is in another language, and when it does, it'll show a sheet at the bottom that says "get translations instantly as you chat. Translations take place on-device and are never sent to Google." If you accept, the following sound effect will be played (provided the Pixel 6 isn't on mute):
[audio ogg="https://static1.xdaimages.com/wordpress/wp-content/uploads/2021/09/auto_translate_inline_translation.ogg"][/audio]
auto_translate_inline_translation.ogg
There will, of course, be an option to turn off translation sounds if you find this annoying. Next, a button to "translate all messages in this chat" will also be shown to speed up translating entire conversations. Translated text will be copied to the phone's clipboard, making it easy to share or forward messages.
If you don't want to translate a message, you can dismiss the bottom sheet, after which you'll be told that you can select and copy a message to start translating again. The feature won't always nag the user about future translations, either, as you have the option to choose whether or not you want to be prompted to translate every time a supported language is detected.
We don't know if this will work in all messaging apps or if it'll only work in preapproved ones, like Google's own Messages app.
Translating text in camera
The Google Camera app will sometimes show a suggestion chip at the bottom, powered by Google Lens, whenever it detects something actionable, like an email address. It sounds like this integration will be improved even further, with Live Translate offering to show text translations in the camera app whenever a supported language is detected. Google Lens can already overlay text translations on top of the viewfinder in real-time, and it can even do it offline, too. However, it'll be convenient to have this feature integrated into Google Camera, if that's indeed what the "Camera" feature implies.
Live Translate will be a useful addition to the Pixel software repertoire. It consolidates the best translation features from Google Translate, Gboard, and Assistant, and it enables all of these features to be used without an active Internet connection. The Pixel 6 series is expected to launch sometime next month, so we'll have to wait a few weeks to see how fast and practical this feature really is.