diff --git a/CHANGELOG.md b/CHANGELOG.md index ed49e18a..db4262a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.8.10] - 2023-11-25 +### Added +- New! Websocket audio ingestion server (optional). Use it to enable audio streaming from a custom desktop client application, from the sample streaming app (below), or from the new LCA Web UI audio streaming client. See [Websocket server](./lca-websocket-stack/README.md). +- New! Sample audio streaming application which streams a stereo audio file from your desktop to LCA via the new websocket server. Use this to replay recordings, e.g. for testing transcription accuracy after CV/CLM changes, for testing agent assist features, etc. Or use the code as a reference implementation for building your own custom streaming app for LCA. See [Websocket client app](./utilities/websocket-client/README.md). +- New! Web UI audio streaming client built into the LCA UI. Use this to stream audio into LCA from (a) your computer microphone, and (b) from a local broswer tab that is running a softphone or meeting app, or playing an audio recording. See [Web UI streaming client](./lca-ai-stack/WebUIStreamingClient.md). +### Fix +- publish.sh script now runs on MacOS (now works with GNU or BSD sed command). + ## [0.8.9] - 2023-11-01 ### Added - Allow customer to specify their own Voice Connector as a CloudFormation Template parameter. If the customer provided Voice Connector is provided, LCA will not deploy a new VC. (#102) @@ -335,8 +343,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Initial release -[Unreleased]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.9...develop -[0.8.9]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.7...v0.8.9 +[Unreleased]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.10...develop +[0.8.10]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.9...v0.8.10 +[0.8.9]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.8...v0.8.9 [0.8.8]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.7...v0.8.8 [0.8.7]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.6...v0.8.7 [0.8.6]: https://github.com/aws-samples/amazon-transcribe-live-call-analytics/compare/v0.8.5...v0.8.6 diff --git a/README.md b/README.md index b4e91dec..385ab474 100644 --- a/README.md +++ b/README.md @@ -102,71 +102,72 @@ To get LCA up and running in your own AWS account, follow these steps (if you do **Telephony Ingestion Options** 4. `Call Audio Source` - Choose `Demo Asterisk PBX Server` to automatically install a demo Asterisk server for testing Amazon Chime SDK Voice Connector streaming 5. `Call Audio Processor` - Choose `Amazon Chime SDK Call Analytics` to use the new Amazon Chime SDK Call Analytics service features instead of the LCA Call transcriber Lambda. See [ChimeCallAnalytics](./lca-chimevc-stack/ChimeCallAnalytics.md). - 6. `Chime Voice Tone Analysis` - Choose only when Amazon Chime SDK Call Analytics is used as the call processor. Enables you to analyze caller voices for a positive, negative, or neutral tone. This is different than sentiment analysis, as it analyzes the audio versus text. --NOTE-- In some jurisdictions, it may not be legal to use voice analytics without the caller's consent. Please read https://docs.aws.amazon.com/chime-sdk/latest/dg/va-opt-out.html for more information. See [ChimeCallAnalytics](./lca-chimevc-stack/ChimeCallAnalytics.md). - 7. `Allowed CIDR Block for Demo Softphone` - Ignored if `Call Audio Source` is not set to `Demo Asterisk PBX Server`. CIDR block allowed by demo Asterisk server for soft phone registration. Example: '10.1.1.0/24' - 8. `Allowed CIDR List for SIPREC Integration` - Ignored if `Call Audio Source ` is not set to `Demo Asterisk PBX Server`. Comma delimited list of CIDR blocks allowed byAmazon Chime SDK Voice Connector for SIPREC source hosts. Example: '10.1.1.0/24, 10.1.2.0/24' - 9. `Lambda Hook Function ARN for SIPREC Call Initialization (existing)` - Used only when CallAudioSource is set to 'Chime Voice Connector (SIPREC)' or 'Demo Asterisk PBX Server'. If present, the specified Lambda function can selectively choose calls to process or to suspend, toggle agent/caller streams, assign AgentId, and/or modify values for CallId and displayed phone numbers. See [LambdaHookFunction.md](./lca-chimevc-stack/LambdaHookFunction.md). - 10. `Amazon Connect instance ARN (existing)` - Ignored if `Call Audio Source ` is not set to `Amazon Connect Contact Lens`. Amazon Connect instance ARN of working instance. Prerequisite: Agent queue and Real Time Contact Lens must be enabled - see [Amazon Connect Integration README](/lca-connect-integration-stack/README.md). + 6. `WebSocketAudioInput` - Enable this option to ingest and analyze audio from the web and microphone. + 7. `Chime Voice Tone Analysis` - Choose only when Amazon Chime SDK Call Analytics is used as the call processor. Enables you to analyze caller voices for a positive, negative, or neutral tone. This is different than sentiment analysis, as it analyzes the audio versus text. --NOTE-- In some jurisdictions, it may not be legal to use voice analytics without the caller's consent. Please read https://docs.aws.amazon.com/chime-sdk/latest/dg/va-opt-out.html for more information. See [ChimeCallAnalytics](./lca-chimevc-stack/ChimeCallAnalytics.md). + 8. `Allowed CIDR Block for Demo Softphone` - Ignored if `Call Audio Source` is not set to `Demo Asterisk PBX Server`. CIDR block allowed by demo Asterisk server for soft phone registration. Example: '10.1.1.0/24' + 9. `Allowed CIDR List for SIPREC Integration` - Ignored if `Call Audio Source ` is not set to `Demo Asterisk PBX Server`. Comma delimited list of CIDR blocks allowed byAmazon Chime SDK Voice Connector for SIPREC source hosts. Example: '10.1.1.0/24, 10.1.2.0/24' + 10. `Lambda Hook Function ARN for SIPREC Call Initialization (existing)` - Used only when CallAudioSource is set to 'Chime Voice Connector (SIPREC)' or 'Demo Asterisk PBX Server'. If present, the specified Lambda function can selectively choose calls to process or to suspend, toggle agent/caller streams, assign AgentId, and/or modify values for CallId and displayed phone numbers. See [LambdaHookFunction.md](./lca-chimevc-stack/LambdaHookFunction.md). + 11. `Amazon Connect instance ARN (existing)` - Ignored if `Call Audio Source ` is not set to `Amazon Connect Contact Lens`. Amazon Connect instance ARN of working instance. Prerequisite: Agent queue and Real Time Contact Lens must be enabled - see [Amazon Connect Integration README](/lca-connect-integration-stack/README.md). **Agent Assist Options** - 11. `Enable Agent Assist` - Choose `QnABot on AWS with new Kendra Index (Developer Edition)` to automatically install all the components and demo configuration needed to experiment with the new Agent Assist capabilities of LCA. See [Agent Assist README](/lca-agentassist-setup-stack/README.md). If you want to integrate LCA with your own agent assist bots or knowledge bases using either Amazon Lex or your own custom implementations, choose `Bring your own LexV2 bot` or `Bring your own AWS Lambda function`. Or choose `Disable` if you do not want any agent assistant capabilities. - 12. `Agent Assist Kendra IndexId (existing)`, `Agent Assist LexV2 BotId (existing)`, `Agent Assist LexV2 Bot AliasId (existing)`, and `Agent Assist Lambda Function ARN (existing)` - empty by default, but must be populated as described depending on the option chosen for `Enable Agent Assist`. - 13. `Agent Assist QnABot Item Matching Api` - Use Kendra for FAQ matching, or choose to use the new QnAbot Embeddings feature - see [QnaBot Embeddings](https://github.com/aws-solutions/qnabot-on-aws/blob/main/docs/semantic_matching_using_LLM_embeddings/README.md) for more information. - 14. `Agent Assist QnABot LLM API` - Use QnABot's (experimental) LLM integration capabilities to use exiting new generative AI capabilities to handle conversational followup questions and to generate concise answers from your knowledge base documents. See [LLM Query Disambiguation and Generative Question Answering](https://github.com/aws-solutions/qnabot-on-aws/blob/develop/docs/LLM_Retrieval_and_generative_question_answering/README.md) - 15. `Agent Assist QnABot Bedrock ModelId` - If `BEDROCK` is chosen for the `Agent ASsist QnABot LLM API`, this is the Bedrock Model ID to use. You must [request model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for the model selected. - 16. `Agent Assist QnABot LLM Lambda Function ARN (existing)` - If LLMApi is `LAMBDA`, provide ARN for a Lambda function that takes JSON {"prompt":"string", "settings":{key:value,..}}, and returns JSON {"generated_text":"string"} - 17. `Agent Assist QnABot LLM Third Party API Key` - If LLMApi is ANTHROPIC, enter your Anthropic API Key. ** Data will leave your AWS account ** + 12. `Enable Agent Assist` - Choose `QnABot on AWS with new Kendra Index (Developer Edition)` to automatically install all the components and demo configuration needed to experiment with the new Agent Assist capabilities of LCA. See [Agent Assist README](/lca-agentassist-setup-stack/README.md). If you want to integrate LCA with your own agent assist bots or knowledge bases using either Amazon Lex or your own custom implementations, choose `Bring your own LexV2 bot` or `Bring your own AWS Lambda function`. Or choose `Disable` if you do not want any agent assistant capabilities. + 13. `Agent Assist Kendra IndexId (existing)`, `Agent Assist LexV2 BotId (existing)`, `Agent Assist LexV2 Bot AliasId (existing)`, and `Agent Assist Lambda Function ARN (existing)` - empty by default, but must be populated as described depending on the option chosen for `Enable Agent Assist`. + 14. `Agent Assist QnABot Item Matching Api` - Use Kendra for FAQ matching, or choose to use the new QnAbot Embeddings feature - see [QnaBot Embeddings](https://github.com/aws-solutions/qnabot-on-aws/blob/main/docs/semantic_matching_using_LLM_embeddings/README.md) for more information. + 15. `Agent Assist QnABot LLM API` - Use QnABot's (experimental) LLM integration capabilities to use exiting new generative AI capabilities to handle conversational followup questions and to generate concise answers from your knowledge base documents. See [LLM Query Disambiguation and Generative Question Answering](https://github.com/aws-solutions/qnabot-on-aws/blob/develop/docs/LLM_Retrieval_and_generative_question_answering/README.md) + 16. `Agent Assist QnABot Bedrock ModelId` - If `BEDROCK` is chosen for the `Agent ASsist QnABot LLM API`, this is the Bedrock Model ID to use. You must [request model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for the model selected. + 17. `Agent Assist QnABot LLM Lambda Function ARN (existing)` - If LLMApi is `LAMBDA`, provide ARN for a Lambda function that takes JSON {"prompt":"string", "settings":{key:value,..}}, and returns JSON {"generated_text":"string"} + 18. `Agent Assist QnABot LLM Third Party API Key` - If LLMApi is ANTHROPIC, enter your Anthropic API Key. ** Data will leave your AWS account ** **Amazon S3 Configuration** - 17. `Call Audio Recordings Bucket Name` - (Optional) Existing bucket where call recording files will be stored. Leave blank to automatically create new bucket - 18. `Audio File Prefix` - The Amazon S3 prefix where the audio files will be saved (must end in "/") - 19. `Call Analytics Output File Prefix` - The Amazon S3 prefix where the post-call analytics files will be saved, when using analytics api mode (must end in "/") + 19. `Call Audio Recordings Bucket Name` - (Optional) Existing bucket where call recording files will be stored. Leave blank to automatically create new bucket + 20. `Audio File Prefix` - The Amazon S3 prefix where the audio files will be saved (must end in "/") + 21. `Call Analytics Output File Prefix` - The Amazon S3 prefix where the post-call analytics files will be saved, when using analytics api mode (must end in "/") **Amazon Transcribe Configuration** - 20. `Enable Partial Transcripts` - Enable partial transcripts to receive low latency evolving transcriptions for each conversation turn. - 21. `Transcribe API mode` - Set the default API mode for Transcribe. Set to 'analytics' to use the Amazon Transcribe Real-time Call Analytics service, used to support call categories and alerts, call summarization, and PCA integration. - 22. `Enable Content Redaction for Transcripts` - Enable content redaction from Amazon Transcribe transcription output. **NOTE:** Content redaction is only available when using the English language (en-US). This parameter is ignored when not using the English language - 23. `Language for Transcription` - Language code to be used for Amazon Transcribe - 24. `Content Redaction Type for Transcription` - Type of content redaction from Amazon Transcribe transcription output - 25. `Transcription PII Redaction Entity Types` - Select the PII entity types you want to identify or redact. Remove the values that you don't want to redact from the default. _DO NOT ADD CUSTOM VALUES HERE_. - 26. `Transcription Custom Vocabulary Name` - The name of the vocabulary to use when processing the transcription job. Leave blank if no custom vocabulary to be used. If yes, the custom vocabulary must pre-exist in your account. - 27. `Transcription Custom Language Model Name` - The name of the custom language model to use when processing the transcription job. Leave blank if no custom language model is to be used. If specified, the custom language model must pre-exist in your account, match the Language Code selected above, and use the 'Narrow Band' base model. + 22. `Enable Partial Transcripts` - Enable partial transcripts to receive low latency evolving transcriptions for each conversation turn. + 23. `Transcribe API mode` - Set the default API mode for Transcribe. Set to 'analytics' to use the Amazon Transcribe Real-time Call Analytics service, used to support call categories and alerts, call summarization, and PCA integration. + 24. `Enable Content Redaction for Transcripts` - Enable content redaction from Amazon Transcribe transcription output. **NOTE:** Content redaction is only available when using the English language (en-US). This parameter is ignored when not using the English language + 25. `Language for Transcription` - Language code to be used for Amazon Transcribe + 26. `Content Redaction Type for Transcription` - Type of content redaction from Amazon Transcribe transcription output + 27. `Transcription PII Redaction Entity Types` - Select the PII entity types you want to identify or redact. Remove the values that you don't want to redact from the default. _DO NOT ADD CUSTOM VALUES HERE_. + 28. `Transcription Custom Vocabulary Name` - The name of the vocabulary to use when processing the transcription job. Leave blank if no custom vocabulary to be used. If yes, the custom vocabulary must pre-exist in your account. + 29. `Transcription Custom Language Model Name` - The name of the custom language model to use when processing the transcription job. Leave blank if no custom language model is to be used. If specified, the custom language model must pre-exist in your account, match the Language Code selected above, and use the 'Narrow Band' base model. **Transcript Event Processing Configuration** - 28. `Enable Sentiment Analysis` - Enable or disable display of sentiment analysis. - 29. `Sentiment Negative Score Threshold` - Minimum negative sentiment confidence required to declare a phrase as having negative sentiment, in the range 0-1. Not applicable when using Contact Lens or Transcribe Call Analytics (as sentiment is pre-calculated). - 30. `Sentiment Positive Score Threshold` - Minimum positive sentiment confidence required to declare a phrase as having positive sentiment, in the range 0-1. Not applicable when using Contact Lens or Transcribe Call Analytics (as sentiment is pre-calculated). - 31. `Lambda Hook Function ARN for Custom Transcript Segment Processing (existing)` - If present, the specified Lambda function is invoked by the LCA Call Event Processor Lambda function for each + 30. `Enable Sentiment Analysis` - Enable or disable display of sentiment analysis. + 31. `Sentiment Negative Score Threshold` - Minimum negative sentiment confidence required to declare a phrase as having negative sentiment, in the range 0-1. Not applicable when using Contact Lens or Transcribe Call Analytics (as sentiment is pre-calculated). + 32. `Sentiment Positive Score Threshold` - Minimum positive sentiment confidence required to declare a phrase as having positive sentiment, in the range 0-1. Not applicable when using Contact Lens or Transcribe Call Analytics (as sentiment is pre-calculated). + 33. `Lambda Hook Function ARN for Custom Transcript Segment Processing (existing)` - If present, the specified Lambda function is invoked by the LCA Call Event Processor Lambda function for each transcript segment. See [TranscriptLambdaHookFunction.md](./lca-ai-stack/TranscriptLambdaHookFunction.md). - 32. `Lambda Hook Function Mode Non-Partial only` - Specifies if Transcript Lambda Hook Function (if specified) is invoked for Non-Partial transcript segments only (true), or for both Partial and Non-Partial transcript segments (false). - 33. `End of Call Transcript Summary` - `BEDROCK` option (default) requires you to choose one of the supported model IDs from the provided list (BedrockModelId). Choose `SAGEMAKER` to automatically deploy a summarization model. Choose `ANTHROPIC` to use the Third Party Anthropic Claude model with your own API key. Alternatively, choose LAMBDA to use your own Lambda function to generate summaries using other models, or choose DISABLED if you are not interested in exploring the new Transcript Summarization feature. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more information. - 34. `BedrockModelId` - If `EndOfCallTranscriptSummary` is `BEDROCK`, then choose a model ID from the list of supported models. Defaults to `anthropic.claude-instant-v1` - 35. `Initial Instance Count for Summarization SageMaker Endpoint` - When `SAGEMAKER` option is chosen (above) enter 0 for a SageMaker Serverless Inference endpoint, or 1 or greater for a provisioned endpoint with the specified number of instances. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. - 36. `End of Call Summarization LLM Third Party API Key` - Provide your API key if you choose ANTHROPIC above. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. - 37. `Lambda Hook Function ARN for Custom End of Call Processing (existing)` - When LAMBDA option is chosen (above) enter the ARN for your custom summarization Lambda function. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. + 34. `Lambda Hook Function Mode Non-Partial only` - Specifies if Transcript Lambda Hook Function (if specified) is invoked for Non-Partial transcript segments only (true), or for both Partial and Non-Partial transcript segments (false). + 35. `End of Call Transcript Summary` - `BEDROCK` option (default) requires you to choose one of the supported model IDs from the provided list (BedrockModelId). Choose `SAGEMAKER` to automatically deploy a summarization model. Choose `ANTHROPIC` to use the Third Party Anthropic Claude model with your own API key. Alternatively, choose LAMBDA to use your own Lambda function to generate summaries using other models, or choose DISABLED if you are not interested in exploring the new Transcript Summarization feature. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more information. + 36. `BedrockModelId` - If `EndOfCallTranscriptSummary` is `BEDROCK`, then choose a model ID from the list of supported models. Defaults to `anthropic.claude-instant-v1` + 37. `Initial Instance Count for Summarization SageMaker Endpoint` - When `SAGEMAKER` option is chosen (above) enter 0 for a SageMaker Serverless Inference endpoint, or 1 or greater for a provisioned endpoint with the specified number of instances. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. + 38. `End of Call Summarization LLM Third Party API Key` - Provide your API key if you choose ANTHROPIC above. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. + 39. `Lambda Hook Function ARN for Custom End of Call Processing (existing)` - When LAMBDA option is chosen (above) enter the ARN for your custom summarization Lambda function. See [Transcript Summarization](./lca-ai-stack/TranscriptSummarization.md) for more details. **Download locations** - 38. `Demo Asterisk Download URL` - (Optional) URL used to download the Asterisk PBX software - 39. `Demo Asterisk Agent Audio URL` - (Optional) + 40. `Demo Asterisk Download URL` - (Optional) URL used to download the Asterisk PBX software + 41. `Demo Asterisk Agent Audio URL` - (Optional) URL for audio (agent.wav) file download for demo Asterisk server. Audio file is automatically played when an agent is not connected with a softphone **Amazon CloudFront Configuration** - 40. `CloudFront Price Class` - The CloudFront price class. See the [CloudFront Pricing](https://aws.amazon.com/cloudfront/pricing/) for a description of each price class. - 41. `CloudFront Allowed Geographies` - (Optional) Comma separated list of two letter country codes (uppercase ISO 3166-1) that are allowed to access the web user interface via CloudFront. For example: US,CA. Leave empty if you do not want geo restrictions to be applied. For details, see: [Restricting the Geographic Distribution of your Content](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/georestrictions.html). + 42. `CloudFront Price Class` - The CloudFront price class. See the [CloudFront Pricing](https://aws.amazon.com/cloudfront/pricing/) for a description of each price class. + 43. `CloudFront Allowed Geographies` - (Optional) Comma separated list of two letter country codes (uppercase ISO 3166-1) that are allowed to access the web user interface via CloudFront. For example: US,CA. Leave empty if you do not want geo restrictions to be applied. For details, see: [Restricting the Geographic Distribution of your Content](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/georestrictions.html). **Record retention** - 42. `Record Expiration In Days` - The length of time, in days, that LCA will retain call records. Records and transcripts that are older than this number of days are permanently deleted. + 44. `Record Expiration In Days` - The length of time, in days, that LCA will retain call records. Records and transcripts that are older than this number of days are permanently deleted. **User Experience** - 43. `Category Alert Regular Expression` - If using the 'analytics' Transcribe API Mode, this regular expression will be used to show an alert in red in the web user interface if it matches a call category. This defaults to matching all categories. + 45. `Category Alert Regular Expression` - If using the 'analytics' Transcribe API Mode, this regular expression will be used to show an alert in red in the web user interface if it matches a call category. This defaults to matching all categories. **Post Call Analytics (PCA) Integration** - 44. `PCA InputBucket` - (Optional) Value of PCA stack "InputBucket". Effective if Transcribe API Mode parameter is 'analytics'. - 45. `PCA InputBucket Transcript prefix` - Value of PCA stack "InputBucketTranscriptPrefix". - 46. `PCA InputBucket Playback AudioFile prefix` - Value of PCA stack "InputBucketPlaybackAudioPrefix". - 47. `PcaWebAppURL` - (Optional) Value of PCA stack "WebAppURL" - allows PCA UI to be launched from LCA UI. - 48. `PCA Web App Call Path Prefix` - PCA path prefix for call detail pages. + 46. `PCA InputBucket` - (Optional) Value of PCA stack "InputBucket". Effective if Transcribe API Mode parameter is 'analytics'. + 47. `PCA InputBucket Transcript prefix` - Value of PCA stack "InputBucketTranscriptPrefix". + 48. `PCA InputBucket Playback AudioFile prefix` - Value of PCA stack "InputBucketPlaybackAudioPrefix". + 49. `PcaWebAppURL` - (Optional) Value of PCA stack "WebAppURL" - allows PCA UI to be launched from LCA UI. + 50. `PCA Web App Call Path Prefix` - PCA path prefix for call detail pages. 5. After reviewing, check the blue box for creating IAM resources. 6. Choose **Create stack**. This will take ~15 minutes to complete. 7. Once the CloudFormation deployment is complete, diff --git a/VERSION b/VERSION index 55485e17..e6663d4c 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.8.9 +0.8.10 \ No newline at end of file diff --git a/images/websocket-start-stream.png b/images/websocket-start-stream.png new file mode 100644 index 00000000..9b5f6cc6 Binary files /dev/null and b/images/websocket-start-stream.png differ diff --git a/images/websocket-stop-stream.png b/images/websocket-stop-stream.png new file mode 100644 index 00000000..813e1237 Binary files /dev/null and b/images/websocket-stop-stream.png differ diff --git a/images/websocket-stream.png b/images/websocket-stream.png new file mode 100644 index 00000000..20c5f242 Binary files /dev/null and b/images/websocket-stream.png differ diff --git a/lca-ai-stack/VERSION b/lca-ai-stack/VERSION index 55485e17..ef505616 100644 --- a/lca-ai-stack/VERSION +++ b/lca-ai-stack/VERSION @@ -1 +1 @@ -0.8.9 +0.8.10 diff --git a/lca-ai-stack/WebUIStreamingClient.md b/lca-ai-stack/WebUIStreamingClient.md new file mode 100644 index 00000000..ac87c531 --- /dev/null +++ b/lca-ai-stack/WebUIStreamingClient.md @@ -0,0 +1,15 @@ +# Web UI Streaming Client + +## Introduction +LCA UI offers an option to stream audio from a browser tab+Microphone to LCA/Agent Assist. +Enable the feature `WebSocketAudioInput` when deploying/updating the LCA stack. + +To use this feature: +1. From the LCA UI, click on `Stream Audio` link from the left navigation menu as shown below. ![Stream](../images/websocket-stream.png) +2. Change default values for Call ID, Agent ID, Customer Phone, and System Phone as needed. Assign a role to the mic input - Agent vs. Caller. +3. Click on `Start Streaming` as shown below.![Stream](../images/websocket-start-stream.png) +4. Share the browser tab that is playing the media (video/audio files, meeting, etc.) +5. Speak into the microphone. +6. The web streaming client combines the audio output from browser tab and the microphone input into a streo (two channel) audio stream. The client sends the stream to the websocket server for downstream processing (enrichment, agent assist, etc.). +7. Click on `Stop Streaming` to end the streaming session. ![Stream](../images/websocket-stop-stream.png) + diff --git a/lca-ai-stack/deployment/lca-ai-stack.yaml b/lca-ai-stack/deployment/lca-ai-stack.yaml index 00333213..2b3eee90 100644 --- a/lca-ai-stack/deployment/lca-ai-stack.yaml +++ b/lca-ai-stack/deployment/lca-ai-stack.yaml @@ -36,6 +36,7 @@ Parameters: - Amazon Chime SDK Voice Connector (SIPREC) - Genesys Cloud Audiohook Web Socket - Amazon Connect Contact Lens + Description: > Choose whether to automatically install a demo Asterisk PBX server for easy standalone testing, a Amazon Chime SDK Voice Connector to use for standards based SIPREC/NBR integration with your contact center, @@ -284,7 +285,7 @@ Parameters: BootstrapVersion: Type: String - Default: 0.8.9 + Default: 0.8.10 Description: > Artifacts version (semver). Used to point to a specific release in the S3 bootstrap bucket @@ -365,13 +366,17 @@ Conditions: Outputs: CallDataStreamName: - Description: >- - The Name of Kinesis Data Stream to write the call data to. - Value: !Ref CallDataStream + Description: >- + The Name of Kinesis Data Stream to write the call data to. + Value: !Ref CallDataStream CallDataStreamArn: Description: >- The ARN of Kinesis Data Stream to write the call data to. Value: !GetAtt CallDataStream.Arn + UserPoolId: + Description: >- + The id of Cognito user pool. + Value: !Ref UserPool S3BucketName: Description: Bucket which contains all the call recordings Value: !If @@ -413,7 +418,10 @@ Outputs: CloudFrontDomainName: Description: The full domain name of the CloudFront distribution Value: !GetAtt WebAppCloudFrontDistribution.DomainName - + LCASettingsParameterName: + Description: Name of the Parameter store + Value: !Ref LCASettingsParameter + TranscriptSummaryFunctionArn: Description: Call Summarizer Lambda function ARN Value: !If [ @@ -545,7 +553,7 @@ Resources: Type: AWS::SSM::Parameter Properties: Type: String - Value: !Sub '{ "CategoryAlertRegex":"${CategoryAlertRegEx}", "EnableVoiceToneAnalysis":"${EnableVoiceToneAnalysis}" }' + Value: !Sub '{ "CategoryAlertRegex":"${CategoryAlertRegEx}", "EnableVoiceToneAnalysis":"${EnableVoiceToneAnalysis}", "WSEndpoint":"" }' ########################################################################## # CodeBuild diff --git a/lca-ai-stack/samconfig.toml b/lca-ai-stack/samconfig.toml index d011b765..e94622e4 100644 --- a/lca-ai-stack/samconfig.toml +++ b/lca-ai-stack/samconfig.toml @@ -8,12 +8,12 @@ use_container = true # shared account [shared.global.parameters] s3_bucket = "lca-artifacts-253873381732-us-east-1" -s3_prefix = "artifacts/lca/0.8.9" +s3_prefix = "artifacts/lca/0.8.10" [shared.deploy.parameters] stack_name = "LiveCallAnalytics" s3_bucket = "lca-artifacts-253873381732-us-east-1" -s3_prefix = "artifacts/lca/0.8.9" +s3_prefix = "artifacts/lca/0.8.10" region = "us-east-1" fail_on_empty_changeset = false confirm_changeset = true @@ -21,7 +21,7 @@ capabilities = "CAPABILITY_IAM CAPABILITY_AUTO_EXPAND" parameter_overrides = [ "BootstrapBucketBaseName=lca-artifacts-253873381732", "BootstrapS3Prefix=artifacts/lca", - "BootstrapVersion=0.8.9", + "BootstrapVersion=0.8.10", "S3BucketName=shared-ai-for-chime-vc-audio", "IsContentRedactionEnabled=true", "IsSentimentAnalysisEnabled=true", @@ -32,4 +32,4 @@ parameter_overrides = [ [shared.package.parameters] s3_bucket = "lca-artifacts-253873381732-us-east-1" -s3_prefix = "artifacts/lca/0.8.9" +s3_prefix = "artifacts/lca/0.8.10" diff --git a/lca-ai-stack/source/ui/package-lock.json b/lca-ai-stack/source/ui/package-lock.json index 4019ccd8..5968f33d 100644 --- a/lca-ai-stack/source/ui/package-lock.json +++ b/lca-ai-stack/source/ui/package-lock.json @@ -1,12 +1,12 @@ { "name": "lca-ui", - "version": "0.8.9", + "version": "0.8.10", "lockfileVersion": 2, "requires": true, "packages": { "": { "name": "lca-ui", - "version": "0.8.9", + "version": "0.8.10", "dependencies": { "@aws-amplify/ui-components": "^1.9.6", "@aws-amplify/ui-react": "^1.2.26", @@ -35,6 +35,7 @@ "react-markdown": "^8.0.3", "react-router-dom": "^5.3.0", "react-scripts": "5.0.1", + "react-use-websocket": "^3.0.0", "rehype-raw": "^6.1.1", "vuera": "^0.2.7", "xlsx": "^0.18.5" @@ -21668,9 +21669,9 @@ } }, "node_modules/caniuse-lite": { - "version": "1.0.30001447", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001447.tgz", - "integrity": "sha512-bdKU1BQDPeEXe9A39xJnGtY0uRq/z5osrnXUw0TcK+EYno45Y+U7QU9HhHEyzvMDffpYadFXi3idnSNkcwLkTw==", + "version": "1.0.30001555", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001555.tgz", + "integrity": "sha512-NzbUFKUnJ3DTcq6YyZB6+qqhfD112uR3uoEnkmfzm2wVzUNsFkU7AwBjKQ654Sp5cau0JxhFyRSn/tQZ+XfygA==", "funding": [ { "type": "opencollective", @@ -21679,6 +21680,10 @@ { "type": "tidelift", "url": "https://tidelift.com/funding/github/npm/caniuse-lite" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" } ] }, @@ -33341,6 +33346,15 @@ "react": "^16.0.0 || ^17.0.0 || ^18.0.0" } }, + "node_modules/react-use-websocket": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/react-use-websocket/-/react-use-websocket-3.0.0.tgz", + "integrity": "sha512-BInlbhXYrODBPKIplDAmI0J1VPM+1KhCLN09o+dzgQ8qMyrYs4t5kEYmCrTqyRuMTmpahylHFZWQXpfYyDkqOw==", + "peerDependencies": { + "react": ">= 16.8.0", + "react-dom": ">= 16.8.0" + } + }, "node_modules/react-virtual": { "version": "2.10.4", "resolved": "https://registry.npmjs.org/react-virtual/-/react-virtual-2.10.4.tgz", @@ -55493,9 +55507,9 @@ } }, "caniuse-lite": { - "version": "1.0.30001447", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001447.tgz", - "integrity": "sha512-bdKU1BQDPeEXe9A39xJnGtY0uRq/z5osrnXUw0TcK+EYno45Y+U7QU9HhHEyzvMDffpYadFXi3idnSNkcwLkTw==" + "version": "1.0.30001555", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001555.tgz", + "integrity": "sha512-NzbUFKUnJ3DTcq6YyZB6+qqhfD112uR3uoEnkmfzm2wVzUNsFkU7AwBjKQ654Sp5cau0JxhFyRSn/tQZ+XfygA==" }, "case-sensitive-paths-webpack-plugin": { "version": "2.4.0", @@ -63912,6 +63926,12 @@ "react-is": "^16.12.0 || ^17.0.0 || ^18.0.0" } }, + "react-use-websocket": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/react-use-websocket/-/react-use-websocket-3.0.0.tgz", + "integrity": "sha512-BInlbhXYrODBPKIplDAmI0J1VPM+1KhCLN09o+dzgQ8qMyrYs4t5kEYmCrTqyRuMTmpahylHFZWQXpfYyDkqOw==", + "requires": {} + }, "react-virtual": { "version": "2.10.4", "resolved": "https://registry.npmjs.org/react-virtual/-/react-virtual-2.10.4.tgz", diff --git a/lca-ai-stack/source/ui/package.json b/lca-ai-stack/source/ui/package.json index eb5a46e2..3ba67013 100644 --- a/lca-ai-stack/source/ui/package.json +++ b/lca-ai-stack/source/ui/package.json @@ -1,6 +1,6 @@ { "name": "lca-ui", - "version": "0.8.9", + "version": "0.8.10", "private": true, "dependencies": { "@aws-amplify/ui-components": "^1.9.6", @@ -30,6 +30,7 @@ "react-markdown": "^8.0.3", "react-router-dom": "^5.3.0", "react-scripts": "5.0.1", + "react-use-websocket": "^3.0.0", "rehype-raw": "^6.1.1", "vuera": "^0.2.7", "xlsx": "^0.18.5" diff --git a/lca-ai-stack/source/ui/public/worklets/recording-processor.js b/lca-ai-stack/source/ui/public/worklets/recording-processor.js new file mode 100644 index 00000000..cbb71cd3 --- /dev/null +++ b/lca-ai-stack/source/ui/public/worklets/recording-processor.js @@ -0,0 +1,113 @@ +// Based on sample from +// https://github.com/GoogleChromeLabs/web-audio-samples/blob/main/src/audio-worklet/migration/worklet-recorder/recording-processor.js + +class RecordingProcessor extends AudioWorkletProcessor { + constructor(options) { + super(); + this.sampleRate = 0; + this.maxRecordingFrames = 0; + this.numberOfChannels = 0; + this._frameSize = 128; + + if (options && options.processorOptions) { + const { + numberOfChannels, + sampleRate, + maxFrameCount, + } = options.processorOptions; + + this.sampleRate = sampleRate; + this.maxRecordingFrames = maxFrameCount; + this.numberOfChannels = numberOfChannels; + } + + this._leftRecordingBuffer = new Float32Array(this.maxRecordingFrames); + this._rightRecordingBuffer = new Float32Array(this.maxRecordingFrames); + + this.recordedFrames = 0; + this.isRecording = false; + + this.framesSinceLastPublish = 0; + this.publishInterval = this.sampleRate * 5; + + this.port.onmessage = (event) => { + if (event.data.message === 'UPDATE_RECORDING_STATE') { + this.isRecording = event.data.setRecording; + } + }; + } + + process(inputs, outputs) { + let currentSample = 0.0; + for (let input = 0; input < 1; input++) { + for (let channel = 0; channel < this.numberOfChannels; channel++) { + for (let sample = 0; sample < inputs[input][channel].length; sample++) { + + currentSample = inputs[input][channel][sample]; + + if (this.isRecording) { + if (channel == 0) { + this._leftRecordingBuffer[sample+this.recordedFrames] = currentSample; + } else if (channel == 1) { + this._rightRecordingBuffer[sample+this.recordedFrames] = currentSample; + } + } + // Pass data directly to output, unchanged. + outputs[input][channel][sample] = currentSample; + } + + } + } + + const shouldPublish = this.framesSinceLastPublish >= this.publishInterval; + + // Validate that recording hasn't reached its limit. + if (this.isRecording) { + if (this.recordedFrames + this._frameSize < this.maxRecordingFrames) { + this.recordedFrames += this._frameSize; + + // Post a recording recording length update on the clock's schedule + if (shouldPublish) { + const recordingBuffer = new Array(this.numberOfChannels) + .fill(new Float32Array(this.maxRecordingFrames)); + recordingBuffer[0] = this._leftRecordingBuffer; + recordingBuffer[1] = this._rightRecordingBuffer; + this.port.postMessage({ + message: 'SHARE_RECORDING_BUFFER', + buffer: recordingBuffer, + recordingLength: this.recordedFrames + }); + this.framesSinceLastPublish = 0; + this.recordedFrames = 0 + } else { + this.framesSinceLastPublish += this._frameSize; + } + } else { + this.recordedFrames += this._frameSize; + + const recordingBuffer = new Array(this.numberOfChannels) + .fill(new Float32Array(this.maxRecordingFrames)); + recordingBuffer[0] = this._leftRecordingBuffer; + recordingBuffer[1] = this._rightRecordingBuffer; + + this.port.postMessage({ + message: 'SHARE_RECORDING_BUFFER', + buffer: recordingBuffer, + recordingLength: this.recordedFrames + }); + + this.recordedFrames = 0; + this.framesSinceLastPublish = 0; + } + } else { + console.log('stopping worklet processor node') + this.recordedFrames = 0; + this.framesSinceLastPublish = 0; + return false; + } + + return true; + } +} + +registerProcessor('recording-processor', RecordingProcessor); diff --git a/lca-ai-stack/source/ui/src/App.jsx b/lca-ai-stack/source/ui/src/App.jsx index f48b5689..1e471966 100644 --- a/lca-ai-stack/source/ui/src/App.jsx +++ b/lca-ai-stack/source/ui/src/App.jsx @@ -22,6 +22,7 @@ const App = () => { const { authState, user } = useUserAuthState(awsConfig); const { currentSession, currentCredentials } = useCurrentSessionCreds({ authState }); const [errorMessage, setErrorMessage] = useState(); + const [navigationOpen, setNavigationOpen] = useState(true); // eslint-disable-next-line react/jsx-no-constructed-context-values const appContextValue = { @@ -32,6 +33,8 @@ const App = () => { currentSession, setErrorMessage, user, + navigationOpen, + setNavigationOpen, }; logger.debug('appContextValue', appContextValue); diff --git a/lca-ai-stack/source/ui/src/components/call-analytics-layout/CallAnalyticsLayout.jsx b/lca-ai-stack/source/ui/src/components/call-analytics-layout/CallAnalyticsLayout.jsx index 94198dbe..3a484e1f 100644 --- a/lca-ai-stack/source/ui/src/components/call-analytics-layout/CallAnalyticsLayout.jsx +++ b/lca-ai-stack/source/ui/src/components/call-analytics-layout/CallAnalyticsLayout.jsx @@ -26,16 +26,18 @@ import { PERIODS_TO_LOAD_STORAGE_KEY, } from '../call-list/calls-table-config'; +import useAppContext from '../../contexts/app'; + const logger = new Logger('CallAnalyticsLayout'); const CallAnalyticsLayout = () => { + const { navigationOpen, setNavigationOpen } = useAppContext(); + const { path } = useRouteMatch(); logger.debug('path', path); const notifications = useNotifications(); const [toolsOpen, setToolsOpen] = useState(false); - const [navigationOpen, setNavigationOpen] = useState(false); - const [selectedItems, setSelectedItems] = useState([]); const getInitialPeriodsToLoad = () => { diff --git a/lca-ai-stack/source/ui/src/components/call-analytics-layout/navigation.jsx b/lca-ai-stack/source/ui/src/components/call-analytics-layout/navigation.jsx index 41a79999..b5bab493 100644 --- a/lca-ai-stack/source/ui/src/components/call-analytics-layout/navigation.jsx +++ b/lca-ai-stack/source/ui/src/components/call-analytics-layout/navigation.jsx @@ -4,11 +4,12 @@ import React from 'react'; import { Route, Switch } from 'react-router-dom'; import { SideNavigation } from '@awsui/components-react'; -import { CALLS_PATH, DEFAULT_PATH } from '../../routes/constants'; +import { CALLS_PATH, STREAM_AUDIO_PATH, DEFAULT_PATH } from '../../routes/constants'; export const callsNavHeader = { text: 'Call Analytics', href: `#${DEFAULT_PATH}` }; export const callsNavItems = [ { type: 'link', text: 'Calls', href: `#${CALLS_PATH}` }, + { type: 'link', text: 'Stream Audio', href: `#${STREAM_AUDIO_PATH}` }, { type: 'section', text: 'Resources', @@ -31,7 +32,8 @@ export const callsNavItems = [ const defaultOnFollowHandler = (ev) => { // XXX keep the locked href for our demo pages - ev.preventDefault(); + // ev.preventDefault(); + console.log(ev); }; /* eslint-disable react/prop-types */ @@ -44,8 +46,8 @@ const Navigation = ({ diff --git a/lca-ai-stack/source/ui/src/components/stream-audio-layout/StreamAudioLayout.jsx b/lca-ai-stack/source/ui/src/components/stream-audio-layout/StreamAudioLayout.jsx new file mode 100644 index 00000000..f3eb59dc --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio-layout/StreamAudioLayout.jsx @@ -0,0 +1,53 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import React, { useState } from 'react'; +import { Switch, Route, useRouteMatch } from 'react-router-dom'; +import { AppLayout, Flashbar } from '@awsui/components-react'; + +import { Logger } from 'aws-amplify'; +import useNotifications from '../../hooks/use-notifications'; + +import StreamAudio from '../stream-audio/StreamAudio'; +import { appLayoutLabels } from '../common/labels'; + +import Navigation from './navigation'; +import Breadcrumbs from './breadcrumbs'; +import ToolsPanel from './tools-panel'; + +import useAppContext from '../../contexts/app'; + +const logger = new Logger('StreamAudioLayout'); + +const StreamAudioLayout = () => { + const { navigationOpen, setNavigationOpen } = useAppContext(); + const { path } = useRouteMatch(); + // console.log(`StreamAudioLayout Path: ${path}`); + logger.info('path ', path); + + const notifications = useNotifications(); + const [toolsOpen, setToolsOpen] = useState(false); + + return ( + } + navigationOpen={navigationOpen} + onNavigationChange={({ detail }) => setNavigationOpen(detail.open)} + breadcrumbs={} + notifications={} + tools={} + toolsOpen={toolsOpen} + onToolsChange={({ detail }) => setToolsOpen(detail.open)} + content={ + + + + + + } + ariaLabels={appLayoutLabels} + /> + ); +}; + +export default StreamAudioLayout; diff --git a/lca-ai-stack/source/ui/src/components/stream-audio-layout/breadcrumbs.jsx b/lca-ai-stack/source/ui/src/components/stream-audio-layout/breadcrumbs.jsx new file mode 100644 index 00000000..a617efbc --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio-layout/breadcrumbs.jsx @@ -0,0 +1,20 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import React from 'react'; +import { Route, Switch, useRouteMatch } from 'react-router-dom'; + +import StreamAudioBreadcrumbs from '../stream-audio/breadcrumbs'; + +const Breadcrumbs = () => { + const { path } = useRouteMatch(); + + return ( + + + + + + ); +}; + +export default Breadcrumbs; diff --git a/lca-ai-stack/source/ui/src/components/stream-audio-layout/index.js b/lca-ai-stack/source/ui/src/components/stream-audio-layout/index.js new file mode 100644 index 00000000..cb207e62 --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio-layout/index.js @@ -0,0 +1,5 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import StreamAudioLayout from './StreamAudioLayout'; + +export default StreamAudioLayout; diff --git a/lca-ai-stack/source/ui/src/components/stream-audio-layout/navigation.jsx b/lca-ai-stack/source/ui/src/components/stream-audio-layout/navigation.jsx new file mode 100644 index 00000000..58adec62 --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio-layout/navigation.jsx @@ -0,0 +1,58 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import React from 'react'; +import { Route, Switch } from 'react-router-dom'; +import { SideNavigation } from '@awsui/components-react'; + +import { CALLS_PATH, DEFAULT_PATH, STREAM_AUDIO_PATH } from '../../routes/constants'; + +export const callsNavHeader = { text: 'Call Analytics', href: `#${DEFAULT_PATH}` }; +export const callsNavItems = [ + { type: 'link', text: 'Calls', href: `#${CALLS_PATH}` }, + { type: 'link', text: 'Stream Audio', href: `#${STREAM_AUDIO_PATH}` }, + { + type: 'section', + text: 'Resources', + items: [ + { + type: 'link', + text: 'Blog Post', + href: 'https://www.amazon.com/live-call-analytics', + external: true, + }, + { + type: 'link', + text: 'Source Code', + href: 'https://github.com/aws-samples/amazon-transcribe-live-call-analytics', + external: true, + }, + ], + }, +]; + +const defaultOnFollowHandler = (ev) => { + // XXX keep the locked href for our demo pages + // ev.preventDefault(); + console.log(ev); +}; + +/* eslint-disable react/prop-types */ +const Navigation = ({ + activeHref = `#${STREAM_AUDIO_PATH}`, + header = callsNavHeader, + items = callsNavItems, + onFollowHandler = defaultOnFollowHandler, +}) => ( + + + + + +); + +export default Navigation; diff --git a/lca-ai-stack/source/ui/src/components/stream-audio-layout/tools-panel.jsx b/lca-ai-stack/source/ui/src/components/stream-audio-layout/tools-panel.jsx new file mode 100644 index 00000000..e6291635 --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio-layout/tools-panel.jsx @@ -0,0 +1,20 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import React from 'react'; +import { Route, Switch, useRouteMatch } from 'react-router-dom'; + +import StreamAudioToolsPanel from '../stream-audio/tools-panel'; + +const ToolsPanel = () => { + const { path } = useRouteMatch(); + + return ( + + + + + + ); +}; + +export default ToolsPanel; diff --git a/lca-ai-stack/source/ui/src/components/stream-audio/StreamAudio.jsx b/lca-ai-stack/source/ui/src/components/stream-audio/StreamAudio.jsx new file mode 100644 index 00000000..e30f92d0 --- /dev/null +++ b/lca-ai-stack/source/ui/src/components/stream-audio/StreamAudio.jsx @@ -0,0 +1,338 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 +import React, { useState, useRef, useCallback, useEffect } from 'react'; + +import { + Form, + FormField, + SpaceBetween, + Container, + Button, + Input, + Header, + ColumnLayout, + Select, +} from '@awsui/components-react'; +import '@awsui/global-styles/index.css'; +import useWebSocket from 'react-use-websocket'; + +import useAppContext from '../../contexts/app'; +import useSettingsContext from '../../contexts/settings'; + +// const TARGET_SAMPLING_RATE = 8000; +let SOURCE_SAMPLING_RATE; + +// export const downsampleBuffer = (buffer, inputSampleRate = 44100, outputSampleRate = 16000) => { +// if (outputSampleRate === inputSampleRate) { +// return buffer; +// } + +// const sampleRateRatio = inputSampleRate / outputSampleRate; +// const newLength = Math.round(buffer.length / sampleRateRatio); +// const result = new Float32Array(newLength); +// let offsetResult = 0; +// let offsetBuffer = 0; + +// while (offsetResult < result.length) { +// const nextOffsetBuffer = Math.round((offsetResult + 1) * sampleRateRatio); +// let accum = 0; +// let count = 0; + +// for (let i = offsetBuffer; i < nextOffsetBuffer && i < buffer.length; i += 1) { +// accum += buffer[i]; +// count += 1; +// } +// result[offsetResult] = accum / count; +// offsetResult += 1; +// offsetBuffer = nextOffsetBuffer; +// } +// return result; +// }; + +const StreamAudio = () => { + const { currentSession } = useAppContext(); + const { settings } = useSettingsContext(); + const JWT_TOKEN = currentSession.getAccessToken().getJwtToken(); + + const [callMetaData, setCallMetaData] = useState({ + callId: crypto.randomUUID(), + agentId: 'AudioStream', + fromNumber: '+9165551234', + toNumber: '+8001112222', + }); + + const [recording, setRecording] = useState(false); + const [streamingStarted, setStreamingStarted] = useState(false); + const [micInputOption, setMicInputOption] = useState({ label: 'AGENT', value: 'agent' }); + + const getSocketUrl = useCallback(() => { + console.log('Trying to resolve websocket url...'); + return new Promise((resolve) => { + if (settings.WSEndpoint) { + console.log(`Resolved Websocket URL to ${settings.WSEndpoint}`); + resolve(settings.WSEndpoint); + } + }); + }, [settings.WSEndpoint]); + + const { sendMessage } = useWebSocket(getSocketUrl, { + queryParams: { + authorization: `Bearer ${JWT_TOKEN}`, + }, + onOpen: (event) => { + console.log(event); + }, + onClose: (event) => { + console.log(event); + }, + onError: (event) => { + console.log(event); + }, + }); + + const handleCallIdChange = (e) => { + setCallMetaData({ + ...callMetaData, + callId: e.detail.value, + }); + }; + + const handleAgentIdChange = (e) => { + setCallMetaData({ + ...callMetaData, + agentId: e.detail.value, + }); + }; + + const handlefromNumberChange = (e) => { + setCallMetaData({ + ...callMetaData, + fromNumber: e.detail.value, + }); + }; + + const handletoNumberChange = (e) => { + setCallMetaData({ + ...callMetaData, + toNumber: e.detail.value, + }); + }; + + const handleMicInputOptionSelection = (e) => { + setMicInputOption(e.detail.selectedOption); + }; + + const audioProcessor = useRef(); + const audioContext = useRef(); + const displayStream = useRef(); + const micStream = useRef(); + const displayAudioSource = useRef(); + const micAudioSource = useRef(); + const channelMerger = useRef(); + const destination = useRef(); + const audioData = useRef(); + + const pcmEncode = (input) => { + const buffer = new ArrayBuffer(input.length * 2); + const view = new DataView(buffer); + for (let i = 0; i < input.length; i += 1) { + const s = Math.max(-1, Math.min(1, input[i])); + view.setInt16(i * 2, s < 0 ? s * 0x8000 : s * 0x7fff, true); + } + return buffer; + }; + + const interleave = (lbuffer, rbuffer) => { + // const leftAudioBuffer = pcmEncode( + // downsampleBuffer(lbuffer, SOURCE_SAMPLING_RATE, TARGET_SAMPLING_RATE), + // ); + const leftAudioBuffer = pcmEncode(lbuffer); + const leftView = new DataView(leftAudioBuffer); + + // const rightAudioBuffer = pcmEncode( + // downsampleBuffer(rbuffer, SOURCE_SAMPLING_RATE, TARGET_SAMPLING_RATE), + // ); + const rightAudioBuffer = pcmEncode(rbuffer); + const rightView = new DataView(rightAudioBuffer); + + const buffer = new ArrayBuffer(leftAudioBuffer.byteLength * 2); + const view = new DataView(buffer); + + for (let i = 0, j = 0; i < leftAudioBuffer.byteLength; i += 2, j += 4) { + view.setInt16(j, leftView.getInt16(i, true), true); + view.setInt16(j + 2, rightView.getInt16(i, true), true); + } + return buffer; + }; + + const stopRecording = async () => { + if (audioProcessor.current) { + audioProcessor.current.port.postMessage({ + message: 'UPDATE_RECORDING_STATE', + setRecording: false, + }); + audioProcessor.current.port.close(); + audioProcessor.current.disconnect(); + } else { + console.log('no media recorder available to stop'); + } + if (streamingStarted && !recording) { + callMetaData.callEvent = 'END'; + sendMessage(JSON.stringify(callMetaData)); + setStreamingStarted(false); + setCallMetaData({ + ...callMetaData, + callId: crypto.randomUUID(), + }); + } + }; + + const startRecording = async () => { + try { + audioContext.current = new window.AudioContext(); + displayStream.current = await window.navigator.mediaDevices.getDisplayMedia({ + video: true, + audio: { + noiseSuppression: true, + autoGainControl: true, + echoCancellation: true, + }, + }); + + micStream.current = await window.navigator.mediaDevices.getUserMedia({ + video: false, + audio: { + noiseSuppression: true, + autoGainControl: true, + echoCancellation: true, + }, + }); + SOURCE_SAMPLING_RATE = audioContext.current.sampleRate; + + // callMetaData.samplingRate = TARGET_SAMPLING_RATE; + callMetaData.samplingRate = SOURCE_SAMPLING_RATE; + + callMetaData.callEvent = 'START'; + sendMessage(JSON.stringify(callMetaData)); + setStreamingStarted(true); + + displayAudioSource.current = audioContext.current.createMediaStreamSource( + new MediaStream([displayStream.current.getAudioTracks()[0]]), + ); + micAudioSource.current = audioContext.current.createMediaStreamSource( + new MediaStream([micStream.current.getAudioTracks()[0]]), + ); + + channelMerger.current = audioContext.current.createChannelMerger(2); + displayAudioSource.current.connect(channelMerger.current, 0, 0); + micAudioSource.current.connect(channelMerger.current, 0, 1); + + try { + await audioContext.current.audioWorklet.addModule('./worklets/recording-processor.js'); + } catch (error) { + console.log(`Add module error ${error}`); + } + + audioProcessor.current = new AudioWorkletNode(audioContext.current, 'recording-processor', { + processorOptions: { + numberOfChannels: 2, + sampleRate: SOURCE_SAMPLING_RATE, + maxFrameCount: (audioContext.current.sampleRate * 1) / 10, + }, + }); + + audioProcessor.current.port.postMessage({ + message: 'UPDATE_RECORDING_STATE', + setRecording: true, + }); + + destination.current = audioContext.current.createMediaStreamDestination(); + channelMerger.current.connect(audioProcessor.current).connect(destination.current); + + audioProcessor.current.port.onmessageerror = (error) => { + console.log(`Error receving message from worklet ${error}`); + }; + + // buffer[0] - display stream, buffer[1] - mic stream + audioProcessor.current.port.onmessage = (event) => { + if (micInputOption.value === 'agent') { + audioData.current = new Uint8Array( + interleave(event.data.buffer[0], event.data.buffer[1]), + ); + } else { + audioData.current = new Uint8Array( + interleave(event.data.buffer[1], event.data.buffer[0]), + ); + } + sendMessage(audioData.current); + }; + } catch (error) { + alert(`An error occurred while recording: ${error}`); + await stopRecording(); + } + }; + + async function toggleRecording() { + if (recording) { + await startRecording(); + } else { + await stopRecording(); + } + } + + useEffect(() => { + toggleRecording(); + }, [recording]); + + const handleRecording = () => { + if (settings.WSEndpoint) { + setRecording(!recording); + } else { + alert('Enable Websocket Audio input to use this feature'); + } + return recording; + }; + + return ( +
e.preventDefault()}> + + + + } + > + Call Meta data}> + + + + + + + + + + + + + + +