Expose `IdentifyMultipleLanguages` in `client-chime-sdk-meetings`

[ravimakhija314]

Describe the feature

We would like the IdentifyMultipleLanguages option to be added to EngineTranscribeSettings for the Chime SDK Meetings StartMeetingTranscriptionCommand API in client-chime-sdk-meetings.

Currently, AWS Chime SDK Meetings live transcription
accepts an EngineTranscribeSettings object that exposes IdentifyLanguage but
not IdentifyMultipleLanguages. The underlying AWS Transcribe streaming
engine already supports IdentifyMultipleLanguages (a single speaker switching languages
mid-utterance — e.g. code-switching), hence we would just like it to be exposed in client-chime-sdk-meetings.

Use Case

I'm currently working on adding live language translation for a tutoring app (https://upchieve.org/) which sometimes serves bilingual students. The app already uses the Chime SDK client-chime-sdk-meetings for audio chat and Chime handles the transcription (e.g. there is no separate call to AWS Transcribe currently). We want to now add language translation in the app, e.g. we plan to use the audio chat transcript to translate appropriately for the student and teacher during live tutoring sessions.

The problem we've encountered is that the Chime SDK does not expose the IdentifyMultipleLanguages option, hence the transcription is done for only one language per participant for the Chime session, which does not support code-switching (e.g. for Spanglish or other language combinations). Hence, we are not able to leverage this for translation downstream, since the incorrect transcription would already be baked in.

In order to support language translation currently, we would have to therefore use a different approach involving changes to the architecture (e.g. sending audio to the backend or to a media insights pipeline), which is currently not the desirable path. However, if IdentifyMultipleLanguages is exposed via the Chime SDK, and Chime outputs transcriptions with a language code per chunk, we can support code-switching with minimal architecture change, which would be a big help.

Note that the same option — IdentifyMultipleLanguages — is already exposed in Chime Media Pipelines (AmazonTranscribeProcessorConfiguration) already. We would prefer to use the other Chime SDK however, since that is much less of an architecture change.

Proposed Solution

No response

Other Information

Code References

You can see that IdentifyLanguage is exposed in the SDK schema, but IdentifyMultipleLanguages is currently missing here:

https://github.com/aws/aws-sdk-js-v3/blob/f8b4987c1d5b52966d536b36200496ea490327b1/clients/client-chime-sdk-meetings/src/schemas/schemas_0.ts

Doc References

You can see IdentifyLanguage is in the Chime SDK API reference, but IdentifyMultipleLanguages is not there. On the other hand, IdentifyMultipleLanguages is listed in the AWS Transcribe API reference, and that's the option for the code-switching case we would like to be exposed in client-chime-sdk-meetings.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

3.1012.0 (however, the code references cited above are from the main branch, hence the issue is still present)

Environment details (OS name and version, etc.)

N/A

[aBurmeseDev]

Hi @ravimakhija314, thanks for reaching out here. What you're asking requires service model update upstream by Chime SDK team. I've submitted your feedback to the team (ref: P469441166), and since there's no action item for our team, we'll close this issue here. Please feel free to contact Chime SDK team directly via AWS Support with reference number shown.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[ravimakhija314]

Hi @aBurmeseDev - there are many support options at the link you provided, some of which say they require a paid support plan. Can you please clarify which specific link to use to contact the Chime SDK team, and whether a paid support plan is going to be required for this? Thanks!

[aBurmeseDev]

@ravimakhija314 - sorry didn't get notified of your reply as the issue was closed. To answer your question, yes you'd need a paid plan to submit a case or get in contact with support team.