Skip to content

wingding12/Voice-AI-Customer-Service-App

BranchesTags

Repository files navigation

Voice AI Customer Service Platform

Next-Generation Customer Support: Intelligent AI agents that seamlessly collaborate with human representatives to deliver exceptional customer experiences at scale.

TypeScript Node.js React PostgreSQL License

Key Features

AI Voice Agent

  • Ultra-low latency: <500ms response time
  • Natural conversations: Handles interruptions naturally
  • Specialized knowledge: Utility industry expert
  • Emergency detection: Auto-escalates critical issues
  • Powered by Retell AI (STT + LLM + TTS unified)

AI Text Chat

  • Context-aware responses using Gemini LLM
  • Knowledge base integration via RAG pipeline
  • Multi-turn conversations with full history
  • Dynamic switching to human agents
  • Unified experience across voice and text

AI Copilot for Agents

  • Real-time suggestions as conversations flow
  • Sentiment analysis detects customer frustration
  • Policy snippets for instant reference
  • Knowledge search across documentation
  • Powered by Google Gemini with pgvector search

Seamless AI↔Human Switching

  • Conference Bridge pattern: No call drops
  • One-click handoff in agent dashboard
  • Customer-initiated: "I want to speak to a human"
  • Agent-initiated: Take over complex cases
  • Full context preservation across all transitions

Enterprise Analytics

  • Real-time dashboard with live metrics
  • Switch tracking and resolution analytics
  • Performance monitoring and SLA tracking
  • Complete audit trail of all interactions
  • Socket.io powered real-time updates

Architecture

Technology Stack

Backend Node.js, Express, TypeScript
Frontend React 18, Vite, TypeScript
Database PostgreSQL + pgvector (vector embeddings)
Cache Redis (sessions + real-time state)
ORM Prisma (type-safe database access)
Real-time Socket.io (WebSocket communication)
AI Services Retell AI (voice), Google Gemini (chat/copilot)
Telephony Telnyx (phone network integration)
Embeddings OpenAI (RAG knowledge base)

High-Level System Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         CUSTOMER LAYER                               │
│                    (Voice Calls + Text Chat)                         │
└─────────────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┴───────────────┐
              │                               │
              ▼                               ▼
    ┌──────────────────┐           ┌──────────────────┐
    │  VOICE CHANNEL   │           │   TEXT CHANNEL   │
    │                  │           │                  │
    │  Telnyx Phone    │           │  Chat Widget     │
    │  Retell AI STT   │           │  Gemini LLM      │
    │  Retell AI TTS   │           │  Context Memory  │
    └────────┬─────────┘           └────────┬─────────┘
             │                              │
             └──────────────┬───────────────┘
                            │
                            ▼
              ┌──────────────────────────┐
              │    BACKEND CORE          │
              │    (Node.js + Express)   │
              │                          │
              │  • Session Manager       │
              │  • Switch Controller     │
              │  • Copilot Engine        │
              │  • RAG Knowledge Base    │
              │  • Analytics Engine      │
              │  • Webhook Handlers      │
              └─────────┬────────────────┘
                        │
          ┌─────────────┼─────────────┐
          │             │             │
          ▼             ▼             ▼
    ┌──────────┐  ┌──────────┐  ┌──────────┐
    │PostgreSQL│  │  Redis   │  │Socket.io │
    │+pgvector │  │ Sessions │  │Real-time │
    └──────────┘  └──────────┘  └─────┬────┘
                                      │
                                      ▼
                        ┌──────────────────────────┐
                        │   AGENT DASHBOARD        │
                        │      (React SPA)         │
                        │                          │
                        │  • Live Transcript View  │
                        │  • AI Copilot Sidebar    │
                        │  • Queue Management      │
                        │  • Control Panel         │
                        │  • Analytics Dashboard   │
                        └──────────────────────────┘

The Conference Bridge Pattern

The approach to seamless handoffs:

┌─────────────────────────────────────────────────────────────┐
│                    CONFERENCE ROOM                          │
│                                                             │
│   ┌──────────┐    ┌──────────┐    ┌──────────┐            │
│   │ CUSTOMER │    │ AI AGENT │    │  HUMAN   │            │
│   │          │    │          │    │   REP    │            │
│   │ Always   │    │ Muted/   │    │ Muted/   │            │
│   │ Active   │    │ Unmuted  │    │ Unmuted  │            │
│   └──────────┘    └──────────┘    └──────────┘            │
│                                                             │
│   SWITCH = Mute one participant, Unmute another            │
│   RESULT = Zero call drops, full context preserved         │
└─────────────────────────────────────────────────────────────┘

Benefits:

  • No call reconnection required
  • No context loss during handoff
  • Sub-second switching time
  • Customer doesn't hear any interruption
  • Scalable to multiple agents per call

🚀 Getting Started

Prerequisites

  • Node.js 18+ and npm 9+
  • Docker Desktop (optional - for PostgreSQL + Redis)
  • API Keys (see Environment Variables section)

Quick Start (3 minutes)

# 1. Clone the repository
git clone <repository-url>
cd Senpilot-Customer-Service-App

# 2. Install dependencies
npm install

# 3. Set up environment variables
cp .env.example .env
# Edit .env with your API keys (see below)

# 4. Start the development servers
npm run dev

The app will start with:

With Docker (Recommended for Full Features)

For persistent data storage and full analytics:

# Start PostgreSQL + Redis containers
docker-compose up -d

# Initialize the database
npm run db:generate
cd packages/database
export DATABASE_URL="postgresql://postgres:postgres@localhost:5433/customer_service?schema=public"
npx prisma migrate dev --name init
npx tsx src/seed.ts
cd ../..

# Start the app
npm run dev

Access the Platform

URL Description
http://localhost:5173 Customer Demo (Chat + Voice)
http://localhost:5173/agent Agent Dashboard
http://localhost:3001/health Backend Health Check

Demo Scenarios

The platform includes 3 pre-built demo scenarios to showcase different use cases:

Scenario Description
💰 High Bill Dispute Customer frustrated about unexpectedly high bill
🚨 Report Gas Leak Emergency situation requiring immediate escalation
🏠 Setup New Service New customer requesting service activation

Click any scenario button in the Chat or Voice interface to start a pre-configured conversation.

⚙️ Configuration

Environment Variables

Create a .env file in the project root. Copy from .env.example and fill in your values:

# ═══════════════════════════════════════════════════════════
# MINIMAL SETUP (Works without Docker)
# ═══════════════════════════════════════════════════════════

# Server Configuration
PORT=3001
NODE_ENV=development
FRONTEND_URL=http://localhost:5173

# Database (Required - but app falls back to in-memory if unavailable)
DATABASE_URL="postgresql://postgres:postgres@localhost:5433/customer_service?schema=public"
REDIS_URL="redis://localhost:6379"

# ═══════════════════════════════════════════════════════════
# AI SERVICES (Add these for full functionality)
# ═══════════════════════════════════════════════════════════

# Google Gemini - Powers text chat + AI copilot
# Get key at: https://makersuite.google.com/
GEMINI_API_KEY=your_gemini_api_key

# Retell AI - Powers voice calls
# Get key at: https://retellai.com
RETELL_API_KEY=your_retell_api_key
RETELL_AGENT_ID=your_retell_agent_id

# ═══════════════════════════════════════════════════════════
# OPTIONAL SERVICES (Enhanced features)
# ═══════════════════════════════════════════════════════════

# Telnyx - Phone number integration
TELNYX_API_KEY=your_telnyx_api_key
TELNYX_CONNECTION_ID=your_connection_id
TELNYX_PHONE_NUMBER=+1234567890

# OpenAI - For RAG embeddings
OPENAI_API_KEY=your_openai_api_key

# Webhooks - For production deployments
WEBHOOK_BASE_URL=https://your-domain.com

Setting Up AI Services

🎙️ Retell AI (Voice Agent)
  1. Sign up at retellai.com
  2. Create a new agent in the dashboard
  3. Configure the agent:
    • Model: gpt-4o-mini or gpt-4
    • Voice: Select from 11labs voices
    • System prompt: Use utility customer service context
  4. Copy your API key and Agent ID to .env

Utility Voice Agent Setup:

# Use our automated setup script
curl -X POST http://localhost:3001/api/voice/agent/create-llm

# This creates an LLM with:
# - Utility-specialized system prompt
# - Emergency gas leak detection
# - Billing/outage/payment knowledge
# - Natural conversation flow
💬 Google Gemini (Text Chat + Copilot)
  1. Get API key from Google AI Studio
  2. Add to .env: GEMINI_API_KEY=your_key
  3. The platform automatically uses Gemini for:
    • Text chat responses (same personality as voice)
    • Agent copilot suggestions
    • Sentiment analysis
    • Context-aware recommendations

No additional setup required - it works out of the box!

Telnyx (Optional - Phone Integration)
  1. Sign up at telnyx.com
  2. Purchase a phone number
  3. Create a TeXML application
  4. Set webhook URL: https://your-domain/webhooks/telnyx
  5. Assign phone number to application
  6. Add credentials to .env

Note: Phone integration is optional. Voice calls also work via browser WebRTC.

Project Structure

customer-service-app/
├── apps/
│   ├── backend/                      # Node.js API Server
│   │   └── src/
│   │       ├── controllers/          # HTTP endpoints & webhooks
│   │       │   ├── chatController.ts       # Text chat API
│   │       │   ├── voiceController.ts      # Voice call management
│   │       │   ├── switchController.ts     # AI↔Human switching
│   │       │   ├── retellController.ts     # Retell webhooks
│   │       │   └── analyticsController.ts  # Metrics & diagnostics
│   │       ├── services/
│   │       │   ├── chat/             # Chat message processing
│   │       │   ├── voice/            # Voice call handling
│   │       │   ├── ai/               # Gemini LLM integration
│   │       │   ├── copilot/          # AI copilot engine
│   │       │   ├── state/            # Redis session management
│   │       │   └── analytics/        # Metrics aggregation
│   │       ├── sockets/
│   │       │   └── agentGateway.ts   # Socket.io real-time events
│   │       └── server.ts             # Express + Socket.io server
│   │
│   └── web-client/                   # React Frontend
│       └── src/
│           ├── components/
│           │   ├── agent-dashboard/  # Agent UI components
│           │   │   ├── QueuePanel.tsx         # Incoming requests queue
│           │   │   ├── LiveTranscript.tsx     # Real-time conversation
│           │   │   ├── SidebarCopilot.tsx     # AI suggestions panel
│           │   │   ├── ChatReplyInput.tsx     # Agent message input
│           │   │   └── ControlPanel.tsx       # Switch/mute controls
│           │   ├── customer-widget/  # Customer-facing UI
│           │   │   ├── ChatWindow.tsx         # Text chat interface
│           │   │   └── CallButton.tsx         # Voice call button
│           │   └── shared/           # Reusable components
│           ├── hooks/
│           │   ├── useCallState.ts            # Call state + Socket.io
│           │   ├── useAgentQueue.ts           # Queue management
│           │   └── useChatSocket.ts           # Chat real-time sync
│           └── pages/
│               ├── AgentPortal.tsx            # Main agent dashboard
│               └── CustomerDemo.tsx           # Customer demo page
│
├── packages/
│   ├── database/                     # Prisma ORM
│   │   ├── prisma/
│   │   │   └── schema.prisma         # Database models
│   │   └── src/
│   │       ├── index.ts              # Prisma client
│   │       └── seed.ts               # Test data seeder
│   │
│   └── shared-types/                 # TypeScript Interfaces
│       └── src/
│           └── index.ts              # Shared types across apps
│
├── docker-compose.yml                # PostgreSQL + Redis
├── package.json                      # Monorepo workspace config
└── .env                              # Environment variables

User Interface

Agent Dashboard

The command center for human representatives:

  • ** Queue Panel** (Left): Live incoming requests with alerts
  • ** Transcript View** (Center): Real-time conversation display
  • ** Copilot Panel** (Right): AI suggestions and knowledge search
  • ** Control Panel** (Bottom): Switch to/from AI, mute, hold, end
  • ** Metrics Footer**: Active calls, resolution times, performance

Customer Widget

Dual-channel customer interface:

  • Text Chat: Instant messaging with AI/human agents
  • Voice Call: Browser-based WebRTC voice calls
  • Seamless Mode Switching: Toggle between chat and voice
  • Status Indicators: AI vs Human agent, connection status

API Reference

Core Endpoints

Endpoint Method Description
/health GET Health check with service status
/api/chat POST Send customer chat message
/api/chat/respond POST Human agent response
/api/chat/switch POST Switch between AI and human
/api/voice/web-call POST Create browser-based voice call
/api/voice/agent GET Get voice agent configuration
/api/switch POST AI↔Human handoff for voice
/api/analytics/dashboard GET Live dashboard metrics
/api/analytics/switches GET Switch analytics by timeframe
/api/copilot/search POST Search knowledge base

Socket.io Events

Client → Server:

  • agent:join - Agent joins their room
  • call:join - Subscribe to call updates
  • call:request_switch - Request AI↔Human switch
  • chat:send_message - Agent sends chat message
  • queue:subscribe - Subscribe to queue updates
  • metrics:subscribe - Subscribe to live metrics

Server → Client:

  • transcript:update - New message in conversation
  • copilot:suggestion - AI suggestion for agent
  • call:state_update - Call mode changed
  • queue:add - New request in queue
  • queue:update - Queue item updated
  • metrics:update - Dashboard metrics refresh

Analytics & Monitoring

Real-time Metrics

The platform tracks comprehensive analytics:

{
  "overview": {
    "totalCalls": 1547,
    "activeCalls": 12,
    "avgDuration": 245,
    "totalSwitches": 289
  },
  "today": {
    "calls": 87,
    "switches": 23,
    "avgDuration": 198
  },
  "modeDistribution": {
    "aiResolved": 1094, // 70.8% AI resolution
    "humanResolved": 312, // 20.2% human only
    "mixed": 141 // 9.1% both
  },
  "switchMetrics": {
    "avgSwitchTime": 1.2, // Seconds
    "topReasons": {
      "CUSTOMER_REQUEST": 152,
      "COMPLEXITY": 89,
      "ESCALATION": 48
    }
  }
}

Performance Monitoring

  • Average handle time (AHT)
  • First response time (FRT)
  • Resolution rate by channel
  • Agent utilization
  • Customer satisfaction proxy metrics
  • Emergency detection accuracy

Testing

Test Scenarios

Scenario Channel Steps
Happy Path Voice Customer inquiry → AI resolves → Call ends
Escalation Voice Customer requests human → Switch → Human resolves
Emergency Voice Gas leak mentioned → Auto-escalate → Emergency team
Text Chat Chat Customer asks question → AI responds → Follow-up
Multi-switch Both AI → Human → AI → Human (stress test)

Running Tests

# Backend API tests
cd apps/backend
npm test

# Frontend component tests
cd apps/web-client
npm test

# E2E tests (full flow)
npm run test:e2e

Use Cases

1. Utility Companies (Primary)

Our specialized domain with pre-built knowledge:

  • Billing inquiries: Explain charges, rate tiers, high bills
  • Payment support: Set up payment plans, financial hardship
  • Outage reporting: Status updates, estimated restoration
  • Service changes: Start/stop/transfer service
  • Emergency response: Gas leak detection and escalation

ROI: 70% AI resolution rate = ~$3M annual savings for 100-agent call center

2. E-Commerce

  • Order tracking and status updates
  • Returns and refund processing
  • Product recommendations
  • VIP customer prioritization
  • Inventory and shipping inquiries

3. Healthcare

  • Appointment scheduling and reminders
  • Insurance verification
  • Prescription refills
  • General health information (non-diagnosis)
  • HIPAA-compliant audit trails

4. Financial Services

  • Account balance and transaction inquiries
  • Fraud detection and reporting
  • Loan/mortgage application support
  • Investment guidance escalation
  • Compliance-ready conversation logs

Security & Compliance

Data Protection

  • All API calls encrypted with TLS 1.3
  • Database encryption at rest
  • Redis session data encrypted
  • PII data masked in logs

Audit & Compliance

  • Complete conversation transcripts stored
  • Switch events logged with timestamps
  • Agent actions tracked
  • GDPR data deletion support
  • Configurable data retention policies

Access Control

  • Agent authentication (planned)
  • Role-based access control (planned)
  • API key rotation support
  • Rate limiting on public endpoints

Development

Troubleshooting

Redis connection refused

The app automatically falls back to in-memory storage. You'll see:

  Redis unavailable - using in-memory storage
   (Start Redis with: docker-compose up -d)

For persistent sessions, start Docker:

docker-compose up -d
Database connection failed

If you see Prisma errors about database connection:

  1. Option A: Start Docker for full database support: 
    docker-compose up -d
npm run db:generate
  2. Option B: Continue without database (analytics will show mock data)
Voice calls not working

Voice calls require Retell AI configuration:

  1. Sign up at retellai.com
  2. Create a voice agent
  3. Add to .env: 
    RETELL_API_KEY=your_key
RETELL_AGENT_ID=your_agent_id
  4. Restart the server
Text chat shows basic responses

For AI-powered responses, add your Gemini API key:

  1. Get key from Google AI Studio
  2. Add to .env: GEMINI_API_KEY=your_key
  3. Restart the server
Port already in use

Kill existing processes:

# Kill backend (port 3001)
lsof -ti:3001 | xargs kill -9

# Kill frontend (port 5173)
lsof -ti:5173 | xargs kill -9

# Restart
npm run dev

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages