[Bug]: Issue with telephony integration

[weixin-a-weixin]

Bug description

When trying to connect telephony account - Twilio, the call unexpectedly ends and the debugger is able to provide the text response but failing to trigger the TTS properly. Can you please let us know if there an issue with our implementation or is it with regards to the project as a whole

Expected behavior

Even if stale message is logged, proper speech output should be produced if the text hasn't been converted to speech via fallbacks

Steps to reproduce

Followed the exact steps provided in rapidaa docs

Rapida version

Logs or traces (optional)

{
  "id": "c3a8d250-cf45-4a25-ba75-7768b83ab50d",
  "time": {
    "seconds": 1774521916,
    "nanos": 562144789
  },
  "current_context": "aa231707-0e7f-48a3-9600-5fcc5e4bd050",
  "packet": "speak_text",
  "reason": "stale_context",
  "type": "discarded"
}

Deployment type

self-hosted

[iamprashant]

@weixin-a-weixin , Seems like you did not created assistant deployment for twilio with stt and tts?

Can you please try debugger deployment to test out if STT, TTS is working perfectly.

[weixin-a-weixin]

configured the assistant deployment for twilio with google for stt and tts, i tried debugger deployment with the same, receiving the same issue

[iamprashant]

@weixin-a-weixin
Please validate following

1. Check the credential json and confirm if the credential is right. / This is not gemini
2. Check Speech to text service is enabled in your gcp account.
3. Select right region for you gcp account when deploying

In playground(debugger) check the events from stt/tts if there is error.

[weixin-a-weixin]

{
"type": "service_account",
"project_id": "YOUR_PROJECT_ID",
"private_key_id": "YOUR_PRIVATE_KEY_ID",
"private_key": "-----BEGIN PRIVATE KEY-----\nYOUR_PRIVATE_KEY_CONTENT\n-----END PRIVATE KEY-----\n",
"client_email": "YOUR_SERVICE_ACCOUNT_EMAIL",
"client_id": "YOUR_CLIENT_ID",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "YOUR_CLIENT_CERT_URL",
"universe_domain": "googleapis.com"
}

this is the json structure we a re using, region is set to global for text-to-speech

STT:

providergoogle-speech-service
rapida.credential_id*************
listen.modellatest_long
listen.regionglobal
listen.languageen-US
listen.threshold0.5
microphone.eos.timeout700
microphone.eos.providersilence_based_eos
microphone.denoising.providerrn_noise
microphone.vad.providersilero_vad
microphone.vad.threshold0.6

TTS:

providergoogle-speech-service
rapida.credential_id*************
speak.voice.iden-US-Neural2-A
speaker.conjunction.boundaries
speaker.conjunction.break240
speaker.pronunciation.dictionaries

[iamprashant]

@weixin-a-weixin Thanks for info,

Please verify ### Check Speech to text service is enabled in your gcp account.

You will see the tab called events. and filterout with stt/tts

[weixin-a-weixin]

I have enabled in gcp account, we have used the same json for other cases which worked normally.
Tried using deepgram, elevenlabs stt and tts, but facing the same issue, attaching the issue when using google as tts

[iamprashant]

@weixin-a-weixin Looks like there is an issue? Can you confirm if you are using docker or build from source?
Your log says the packet is getting discarded. Might be problem in End-Of-Speech configuration.

[iyerpriyanka]

Could be because of libopus0 libopusfile0 libc++1 had same issue. @iamprashant i think documentation need to be updated

[weixin-a-weixin]

i am using docker, i was testing using the debugger by texting, since the voice option was redirecting to a new page but wasnt loading, after i input some text, this is the flow

when i try accessing voice

[iamprashant]

Thanks for sharing @weixin-a-weixin.

Clear, now there is no packet from WebRTC.
Please update docker/assistant-api/.assistant.env

# WebRTC ICE configuration (required on EC2 / any cloud server)
# EC2 exposes a public IP via AWS NAT but Pion only sees the private 10.x.x.x IP.
# Set EXTERNAL_IP to the EC2 instance's Elastic/public IP so Pion advertises
# the correct address in ICE candidates. Leave empty for local development.
# Also open UDP 20000-20100 inbound in the EC2 security group.
WEBRTC__EXTERNAL_IP={PUBLIC_IP}}
WEBRTC__UDP_PORT_RANGE_START=20000
WEBRTC__UDP_PORT_RANGE_END=20100

In case you are running on aws open security group
port range 20000-20100 UDP

For twilio and other telephony with websocket

# ngork tunnel for local development, needed for callback URLs in external services like Twilio, Google Dialogflow, etc.
PUBLIC_ASSISTANT_HOST=integral-presently-cub.ngrok-free.app

You can find SIP config as well to update if you are connecting with SIP.
Incase it doesn't work please send me a note ps@rapida.ai

[weixin-a-weixin]

I am already using my IP in WEBRTC__EXTERNAL_IP, my provider is not AWS. an ngrok was setup too. Now the call is getting connected but the channel is not receiving or producing audio, and the call is being terminated in under 30 secs. I have repeatedly tested my google, elevenlabs and deepgram credentials, which are working normally.

[iamprashant]

@weixin-a-weixin Can you please share log of assistant-api.
Please join our discord https://discord.gg/ZTZPsxD4St will connect and resolve the issue.