A robust API service for creating conversational AI assistants that can answer questions based on provided documentation and additional context.
This service powers the chatbot features of the CincoData website
ONLY serve this internally (the intended use environment for the chatbot feature), there are no rate limits nor security checks (you can run arbitary REST requests with this 💀)
FAQ-LLM Bot provides a stateful conversation API that:
- Manages multiple independent conversation sessions
- Supports streaming responses for real-time interactions
- Allows dynamic addition of contextual information
- Provides a RESTful interface for easy integration
The service is built on FastAPI and leverages OpenAI's chat completion models to generate responses, with support for custom prompts and conversation management.
├── document_qna/ # Core QnA functionality
│ ├── __init__.py
│ └── qna.py # Main QnA class for managing conversations
├── http_endpoint_types.py # Pydantic models for request/response validation
├── http_endpoints.py # FastAPI endpoints implementation
└── rest_helper.py # Utility for making external REST calls
- Python 3.8+
- OpenAI API key
- Clone the repository
- Install dependencies:
pip install -r requirements.txt
- Create a
.envfile with the following variables:OPENAI_API_KEY=your_api_key OPENAI_MODEL=anthropic/claude-3.7-sonnet:beta # or other model LOG_LEVEL=INFO HTTP_HOST=0.0.0.0 HTTP_PORT=8000 MOUNT_PREFIX= # optional prefix for all endpoints CORS=* # comma-separated list of allowed origins
python http_endpoints.pyThe service will start on the configured port (default: 8000).
Documentation (automatically generated by swaggerUI) are availiable under the /docs endpoint.
-
Initialize a Session
POST /v1/init Content-Type: application/json { "clientArgs": { "api_key": "your_openai_api_key", # Optional, can use env var "baseUrl": "https://api.openai.com/v1" # Optional }, "qnaArgs": { "prompt": "You are a helpful assistant...", # Optional, uses default if not provided "model": "gpt-3.5-turbo", # Optional "temperature": 0.7, # Optional "additionals": { # Optional "faq_content": "Q: What are your hours?\nA: We're open 9-5..." } } }
Response:
{ "payload": { "sessionId": "550e8400-e29b-41d4-a716-446655440000" }, "metadata": { "messageID": "123e4567-e89b-12d3-a456-426614174000", "timestamp": "2024-01-20T12:34:56.789Z" } } -
End a Session
DELETE /v1/end X-Session-ID: 550e8400-e29b-41d4-a716-446655440000
-
Send a Message (Non-streaming)
POST /v1/message X-Session-ID: 550e8400-e29b-41d4-a716-446655440000 Content-Type: application/json { "message": "What are your business hours?", "stream": false }
Response:
{ "payload": { "message": "Our business hours are 9 AM to 5 PM, Monday through Friday." }, "metadata": { "messageID": "123e4567-e89b-12d3-a456-426614174000", "timestamp": "2024-01-20T12:34:56.789Z" } } -
Create a Streaming Job
POST /v1/stream X-Session-ID: 550e8400-e29b-41d4-a716-446655440000 Content-Type: application/json { "message": "Tell me about your return policy." }
Response:
{ "payload": { "jobId": "job-123e4567-e89b-12d3-a456-426614174000" }, "metadata": { "messageID": "123e4567-e89b-12d3-a456-426614174000", "timestamp": "2024-01-20T12:34:56.789Z" } } -
Consume Streaming Response
GET /v1/stream/job-123e4567-e89b-12d3-a456-426614174000 X-Session-ID: 550e8400-e29b-41d4-a716-446655440000
Response: A streaming text response with chunks of the assistant's message.
-
Add Information
POST /v1/additionals X-Session-ID: 550e8400-e29b-41d4-a716-446655440000 Content-Type: application/json { "items": [ { "id": "pricing_info", "description": "Current pricing information", "content": "Basic plan: $10/month\nPro plan: $25/month" }, { "id": "api_data", "description": "Latest API documentation", "content": { "baseUrl": "https://api.example.com/docs", "method": "GET", "headers": { "Authorization": "Bearer token123" } } } ] }
Response:
{ "payload": { "addedItems": ["pricing_info", "api_data"], "message": "Successfully added 2 additional information items" }, "metadata": { "messageID": "123e4567-e89b-12d3-a456-426614174000", "timestamp": "2024-01-20T12:34:56.789Z" } } -
Remove Information
DELETE /v1/additionals X-Session-ID: 550e8400-e29b-41d4-a716-446655440000 Content-Type: application/json { "ids": ["pricing_info"] }
Response:
{ "payload": { "removedItems": ["pricing_info"], "message": "Successfully removed 1 additional information items" }, "metadata": { "messageID": "123e4567-e89b-12d3-a456-426614174000", "timestamp": "2024-01-20T12:34:56.789Z" } }
GET /healthResponse:
{
"status": "healthy",
"error": null,
"version": "1.0.0",
"timestamp": "2024-01-20T12:34:56.789Z",
"metrics": {
"activeSessions": 5,
"lockedSessions": 1
}
}The service maintains stateful sessions, each with its own conversation history and context. Sessions are identified by UUIDs and are accessed via the X-Session-ID header.
Each session has a lock to prevent concurrent modifications, ensuring that only one request can modify a session at a time.
Responses can be streamed in real-time, allowing for more interactive user experiences.
Additional information can be added to a session at any time, either directly or by fetching from external APIs.
The core of the system is the QnA class, which:
- Manages conversation history
- Handles message processing
- Supports streaming responses
- Manages additional context information
The API endpoints provide a RESTful interface to the QnA functionality, with proper validation, error handling, and documentation.
The service includes comprehensive error handling, with appropriate HTTP status codes and error messages.
- Scalability: The service can handle multiple concurrent sessions, but for high-load scenarios, consider deploying multiple instances behind a load balancer.
- Security: Protect the API with appropriate authentication and rate limiting.
- Monitoring: Use the
/healthendpoint to monitor the service's status and metrics.
Contributions are welcome! Please feel free to submit a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.