Vitlycare Outbound Voice Agent (Twilio + Pipecat)

An outbound voice agent for Vitlycare that places phone calls via Twilio, streams audio over Twilio Media Streams, and runs a real-time conversational AI pipeline using Pipecat.

The agent:

• automatically greets the callee on pickup
• listens and transcribes speech using Deepgram STT
• generates responses using OpenAI
• speaks back using Deepgram TTS
• supports interruption and natural turn-taking

This project is intended for demo and early-stage production testing of outbound healthcare calls (e.g. discharge planning).

---

Architecture Overview

Client → POST /dialout
        ↓
Twilio REST API → outbound phone call
        ↓
Twilio fetches /twiml
        ↓
<Connect><Stream> → WebSocket (/ws)
        ↓
Twilio Media Streams
        ↓
Pipecat Pipeline
   STT (Deepgram)
 → LLM (OpenAI)
 → TTS (Deepgram)
        ↓
Audio back to callee

Separation of concerns:

• server.py → telephony & WebSocket plumbing
• bot.py → conversational intelligence
• server_utils.py → Twilio + TwiML helpers

---

Tech Stack

• Telephony: Twilio (Outbound Calls + Media Streams)
• Orchestration: Pipecat
• Speech-to-Text: Deepgram
• Text-to-Speech: Deepgram (Aura voices)
• LLM: OpenAI
• Server: FastAPI
• Runtime: Python 3.10+

---

Repository Structure

.
├── bot.py
├── server.py
├── server_utils.py
├── pyproject.toml
├── env.example
└── README.md

---

Setup

1. Environment

uv venv .venv
source .venv/bin/activate
uv sync

2. Environment Variables

cp env.example .env

Required:

OPENAI_API_KEY=
DEEPGRAM_API_KEY=
TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_PHONE_NUMBER=
ENV=local
LOCAL_SERVER_URL=https://xxxx.ngrok-free.app

---

Running

uv run server.py
ngrok http 7860

---

Make an Outbound Call

curl -X POST https://<ngrok-url>/dialout   -H "Content-Type: application/json"   -d '{
    "to_number": "+91XXXXXXXXXX",
    "from_number": "+1XXXXXXXXXX"
  }'

---

License

BSD 2-Clause License