Google is preparing to let you use “Live Captions” during phone calls

on
Google first announced its “Live Captions” accessibility feature at Google I/O 2019 last year. Using three on-device machine learning models, this feature detects English-language speech in media and generates captions automatically. The feature was first made available on the Pixel 4 but later expanded to the Pixel 3a, Pixel 3, Pixel 2, Samsung Galaxy S20, OnePlus 7T, and OnePlus 8. Since the feature launched, however, it hasn’t received any updates to its functionality. That could change soon, though, as we spotted evidence that Google will allow Live Caption to work over phone calls.
An APK teardown can often predict features that may arrive in a future update of an application, but it is possible that any of the features we mention here may not make it in a future release. This is because these features are currently unimplemented in the live build and may be pulled at any time by the developers in a future build.
In Android 11 Developer Preview 3 for the Google Pixel 4, we analyzed version 2.13.302920511 of Device Personalization Services, the application responsible for Live Captions. In it, we spotted strings that state users will be able to turn on Live Caption over a phone call.
<string name="call_confirmation_cancel">CANCEL</string>
<string name="call_confirmation_confirmation_message">Enable Live Caption over this phone call? Your use of the feature will be annonunced to others on the call.</string>
<string name="call_confirmation_dialog_name">Enable Live Caption confirmation dialog</string>
<string name="call_confirmation_enable">ENABLE</string>
<string name="call_other_side_transcription_prefix">Caller</string>
<string name="call_system_message_prefix">System</string>
<string name="call_turn_indicator_text">…</string>
<string name="call_user_typed_input_prefix">You typed</string>
Enabling the feature over a phone call will announce to all participants that the call is being transcribed into captions. An audio file will be played in the phone call, and this audio file simply says the following line:
<string name="system_message_start_call_speaking_mode">Hi, the person you’re about to speak with has call captions turned on. They’ll see captions of what you say to help them listen along.</string>
Currently, the API that Live Captions relies on, AudioPlaybackCaptureConfiguration, does not allow for capturing voice call audio. However, it’s possible that a new system-only permission added in Android 11 Developer Preview 3 will allow the feature to bypass this restriction.

Comments

Post a Comment