Skip to content

Commit dcd78d9

Browse files
benjigiffordbenjigifford
authored
Merge branch 'main' into add-milvus-test
2 parents 7e4633c + 622ba84 commit dcd78d9

32 files changed

Lines changed: 4278 additions & 183 deletions

‎.env.test‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
1-
CONSERVER_API_TOKEN=fake-token
1+
CONSERVER_API_TOKEN=fake-token
2+
DEEPGRAM_KEY=eb223fe1b47d1b26c6b35485ee0a29c2bc4cd1bb

‎.github/workflows/docker-publish.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: Build and Push Docker image
2+
3+
on:
4+
push:
5+
branches: [main, release-test]
6+
tags: ['v*']
7+
8+
jobs:
9+
build:
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- name: Checkout code
14+
uses: actions/checkout@v4
15+
16+
- name: Set up Docker Buildx
17+
uses: docker/setup-buildx-action@v3
18+
19+
- name: Log in to Docker Hub
20+
uses: docker/login-action@v3
21+
with:
22+
username: ${{ secrets.DOCKER_USERNAME }}
23+
password: ${{ secrets.DOCKER_PASSWORD }}
24+
25+
- name: Build and push Docker image
26+
uses: docker/build-push-action@v5
27+
with:
28+
context: .
29+
file: ./docker/Dockerfile
30+
push: true
31+
tags: |
32+
howethomas/conserver:latest
33+
howethomas/conserver:${{ github.sha }}
34+
# Optionally, add a tag for releases
35+
# howethomas/conserver:${{ github.ref_name }}

‎README.md‎

Lines changed: 35 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ vCon Server is a powerful conversation processing and storage system that enable
2424
- [Monitoring and Logging](#monitoring-and-logging)
2525
- [Troubleshooting](#troubleshooting)
2626
- [License](#license)
27+
- [Production Deployment Best Practices](#production-deployment-best-practices)
28+
- [Example Directory Layout](#example-directory-layout)
29+
- [Example Redis Volume in docker-compose.yml](#example-redis-volume-in-docker-composeyml)
30+
- [User Creation and Permissions](#user-creation-and-permissions)
2731

2832
## Prerequisites
2933

@@ -119,8 +123,8 @@ links:
119123
options:
120124
webhook-urls:
121125
- https://example.com/conserver
122-
deepgram:
123-
module: links.deepgram
126+
deepgram_link:
127+
module: links.deepgram_link
124128
options:
125129
DEEPGRAM_KEY: your_deepgram_key
126130
minimum_duration: 30
@@ -155,7 +159,7 @@ storages:
155159
chains:
156160
main_chain:
157161
links:
158-
- deepgram
162+
- deepgram_link
159163
- summarize
160164
- webhook_store_call_log
161165
storages:
@@ -289,3 +293,31 @@ docker compose logs -f [service_name]
289293
## License
290294

291295
This project is licensed under the terms specified in the LICENSE file.
296+
297+
## Production Deployment Best Practices
298+
299+
- **Install as a non-root user**: Create a dedicated user (e.g., `vcon`) for running the application and Docker containers.
300+
- **Clone repositories to /opt**: Place `vcon-admin` and `vcon-server` in `/opt` for system-wide, non-root access.
301+
- **Use persistent Docker volumes**: Map Redis and other stateful service data to `/opt/vcon-data` for durability.
302+
- **Follow the updated install script**: Use `scripts/install_conserver.sh` which now implements these best practices.
303+
304+
### Example Directory Layout
305+
306+
```
307+
/opt/vcon-admin
308+
/opt/vcon-server
309+
/opt/vcon-data/redis
310+
```
311+
312+
### Example Redis Volume in docker-compose.yml
313+
314+
```yaml
315+
volumes:
316+
- /opt/vcon-data/redis:/data
317+
```
318+
319+
### User Creation and Permissions
320+
321+
The install script creates the `vcon` user and sets permissions for all necessary directories.
322+
323+
---

‎docker-compose.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ services:
5757
REDIS_ARGS: --save 20 1 --notify-keyspace-events Ex --dir /data --appendonly yes
5858
mem_limit: 1gb # <===== IMPORTANT!!!! We're overriding this in the docker-compose.override.yml file
5959
volumes:
60-
- ./.data:/data
60+
- /opt/vcon-data/redis:/data # Production: use a persistent host directory
6161
healthcheck:
6262
test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
6363
interval: 30s

‎prod_mgt/01_PRODUCT_OVERVIEW.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# vCon Server Product Overview
2+
3+
## Executive Summary
4+
5+
vCon Server is an enterprise-grade conversation processing and storage system that provides a complete platform for managing Voice Conversation (vCon) records. It implements a flexible, modular pipeline architecture for processing, analyzing, storing, and retrieving conversation data through various specialized modules and integrations.
6+
7+
## Core Product Vision
8+
9+
vCon Server serves as a centralized hub for conversation data management, enabling organizations to:
10+
- Capture and process conversations from multiple sources
11+
- Apply AI-powered analysis and transcription
12+
- Store conversations in various backend systems
13+
- Query and retrieve conversations efficiently
14+
- Integrate with external systems through webhooks and APIs
15+
16+
## Key Product Components
17+
18+
### 1. Core vCon System
19+
- **vCon Data Model**: Standardized conversation representation format
20+
- **UUID-based Identification**: Unique identifiers for every conversation
21+
- **Metadata Management**: Parties, attachments, analysis, and dialog tracking
22+
- **Tag System**: Flexible tagging for categorization and routing
23+
24+
### 2. Processing Pipeline
25+
- **Chain-based Architecture**: Configurable processing chains
26+
- **Link Processors**: Modular components for specific processing tasks
27+
- **Storage Adapters**: Multiple backend storage options
28+
- **Queue Management**: Redis-based queue system for reliable processing
29+
30+
### 3. API Layer
31+
- **RESTful API**: Complete CRUD operations for vCons
32+
- **Authentication**: Token-based API security
33+
- **Search Capabilities**: Query by phone, email, name, and metadata
34+
- **Configuration Management**: Dynamic system configuration
35+
36+
### 4. Caching & Performance
37+
- **Redis Caching**: Primary cache with configurable expiration
38+
- **Multi-tier Storage**: Fallback to persistent storage backends
39+
- **Indexed Search**: Efficient retrieval via sorted sets
40+
- **Automatic Cache Population**: Smart caching from storage backends
41+
42+
## Target Users
43+
44+
1. **Call Centers & Contact Centers**
45+
- Recording and analyzing customer interactions
46+
- Quality assurance and compliance
47+
- Training and performance improvement
48+
49+
2. **Healthcare Organizations**
50+
- Patient conversation documentation
51+
- Telehealth session management
52+
- Compliance and audit trails
53+
54+
3. **Financial Services**
55+
- Transaction verification
56+
- Compliance recording
57+
- Customer service optimization
58+
59+
4. **Enterprise Communications**
60+
- Meeting transcription and analysis
61+
- Knowledge management
62+
- Collaboration insights
63+
64+
## Key Differentiators
65+
66+
1. **Modular Architecture**: Plug-and-play components for customization
67+
2. **Multi-Storage Support**: Choose from 10+ storage backends
68+
3. **AI Integration**: Built-in support for multiple AI services
69+
4. **Scalability**: Horizontal scaling through containerization
70+
5. **Open Standards**: Based on vCon specification for interoperability
71+
72+
## Deployment Models
73+
74+
1. **On-Premises**: Full control and data sovereignty
75+
2. **Cloud**: Scalable deployment on AWS, Azure, or GCP
76+
3. **Hybrid**: Mix of on-premises and cloud components
77+
4. **Edge**: Distributed processing at edge locations
78+
79+
## Security & Compliance
80+
81+
- Token-based API authentication
82+
- Encryption support at rest and in transit
83+
- Audit trails and access logging
84+
- GDPR-compliant data handling options
85+
- Configurable retention policies

‎prod_mgt/02_CORE_FUNCTIONALITY.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Core Functionality
2+
3+
## vCon Data Model
4+
5+
### Structure
6+
The vCon (Voice Conversation) is the fundamental data structure that represents a conversation:
7+
8+
```json
9+
{
10+
"uuid": "unique-identifier",
11+
"vcon": "0.0.1",
12+
"created_at": "ISO-8601-timestamp",
13+
"redacted": {},
14+
"group": [],
15+
"parties": [
16+
{
17+
"tel": "+1234567890",
18+
"mailto": "user@example.com",
19+
"name": "John Doe",
20+
"role": "agent"
21+
}
22+
],
23+
"dialog": [
24+
{
25+
"type": "recording",
26+
"start": "timestamp",
27+
"duration": 300,
28+
"parties": [0],
29+
"url": "https://example.com/recording.mp3"
30+
}
31+
],
32+
"attachments": [
33+
{
34+
"type": "transcript",
35+
"body": "conversation text...",
36+
"encoding": "none"
37+
}
38+
],
39+
"analysis": [
40+
{
41+
"type": "summary",
42+
"dialog": [0],
43+
"vendor": "openai",
44+
"body": "analysis results..."
45+
}
46+
]
47+
}
48+
```
49+
50+
### Key Components
51+
52+
1. **Parties**: Participants in the conversation
53+
- Phone numbers, emails, names
54+
- Roles (agent, customer, etc.)
55+
- Indexed for efficient searching
56+
57+
2. **Dialog**: The actual conversation content
58+
- Recordings with URLs
59+
- Text messages
60+
- Timestamps and duration
61+
62+
3. **Attachments**: Additional data
63+
- Transcripts
64+
- Metadata
65+
- Tags
66+
- Custom data
67+
68+
4. **Analysis**: AI-generated insights
69+
- Summaries
70+
- Sentiment analysis
71+
- Custom analysis types
72+
- Vendor attribution
73+
74+
## Processing Pipeline
75+
76+
### Chain Architecture
77+
- **Ingress Lists**: Entry points for vCons
78+
- **Processing Links**: Sequential processing steps
79+
- **Storage Operations**: Persistence to backends
80+
- **Egress Lists**: Output queues for processed vCons
81+
82+
### Chain Configuration Example
83+
```yaml
84+
chains:
85+
main_chain:
86+
links:
87+
- deepgram_link # Transcription
88+
- analyze # AI analysis
89+
- tag # Tagging
90+
- webhook # External notification
91+
storages:
92+
- postgres # Primary storage
93+
- s3 # Backup storage
94+
ingress_lists:
95+
- main_ingress
96+
egress_lists:
97+
- processed_queue
98+
enabled: 1
99+
```
100+
101+
### Processing Flow
102+
1. vCon enters via ingress list
103+
2. Each link processes sequentially
104+
3. Storage operations execute
105+
4. vCon moves to egress lists
106+
5. Dead letter queue for failures
107+
108+
## API Functionality
109+
110+
### Core Endpoints
111+
112+
#### vCon Management
113+
- `POST /api/vcon` - Create new vCon
114+
- `GET /api/vcon/{uuid}` - Retrieve vCon
115+
- `PUT /api/vcon/{uuid}` - Update vCon
116+
- `DELETE /api/vcon/{uuid}` - Delete vCon
117+
- `GET /api/vcons` - Batch retrieval
118+
119+
#### Search Operations
120+
- `GET /api/vcons/search?tel={phone}` - Search by phone
121+
- `GET /api/vcons/search?mailto={email}` - Search by email
122+
- `GET /api/vcons/search?name={name}` - Search by name
123+
- Multiple parameters for AND operations
124+
125+
#### Chain Management
126+
- `POST /api/chain/{chain_name}/ingress` - Add to chain
127+
- `POST /api/chain/{chain_name}/egress` - Process chain
128+
129+
#### Configuration
130+
- `GET /api/config` - Get configuration
131+
- `POST /api/config` - Update configuration
132+
133+
#### Queue Management
134+
- `GET /api/dlq` - View dead letter queue
135+
- `POST /api/dlq/reprocess` - Reprocess failed items
136+
137+
### Authentication
138+
- Header-based: `x-conserver-api-token`
139+
- File-based token management
140+
- Environment variable support
141+
142+
## Redis Integration
143+
144+
### Caching Strategy
145+
1. **Primary Storage**: vCons stored as JSON
146+
2. **Sorted Sets**: Timestamp-based ordering
147+
3. **Index Sets**: Search acceleration
148+
4. **Expiration**: Configurable TTL
149+
150+
### Key Patterns
151+
- `vcon:{uuid}` - vCon data
152+
- `tel:{number}` - Phone index
153+
- `mailto:{email}` - Email index
154+
- `name:{name}` - Name index
155+
- `vcons` - Sorted set by timestamp
156+
157+
### Performance Features
158+
- Automatic cache population from storage
159+
- Configurable expiration (VCON_REDIS_EXPIRY)
160+
- Index expiration (VCON_INDEX_EXPIRY)
161+
- Batch operations support
162+
163+
## Error Handling
164+
165+
### Dead Letter Queue (DLQ)
166+
- Automatic failure capture
167+
- Per-ingress-list DLQs
168+
- Reprocessing capability
169+
- Error tracking
170+
171+
### Retry Mechanisms
172+
- Configurable retry counts
173+
- Exponential backoff
174+
- Link-specific retry logic
175+
- Metrics tracking
176+
177+
## Metrics & Monitoring
178+
179+
### Built-in Metrics
180+
- Processing time per link
181+
- Success/failure rates
182+
- Storage operation timing
183+
- API request metrics
184+
185+
### Integration Points
186+
- Datadog support
187+
- Custom metrics via StatsD
188+
- Structured logging
189+
- Error tracking services

0 commit comments

Comments
 (0)