Merge pull request #111 from Rajaniraiyn/sql

Add SQL Conversation Storage

Author: cornelcroi

docs/src/content/docs/storage/dynamodb.mdx

@@ -17,6 +17,14 @@ DynamoDB storage provides a scalable and persistent solution for storing convers
 - When long-term persistence of conversation history is required
 - For applications that need to scale horizontally
 
+## Python Package
+
+If you haven't already installed the AWS-related dependencies, make sure to install them:
+
+```bash
+pip install "multi-agent-orchestrator[aws]"
+```
+
 ## Implementation
 
 To use DynamoDB storage in your Multi-Agent Orchestrator:

docs/src/content/docs/storage/overview.md

@@ -21,19 +21,25 @@ The Multi-Agent Orchestrator System offers flexible storage options for maintain
    - Provides persistent storage for production environments.
    - Allows for scalable and durable conversation history storage.
 
-3. **Custom Storage Solutions**:
+3. **SQL Storage**:
+    - Offers persistent storage using SQLite or Turso databases.
+    - When you need local-first development with remote deployment options
+
+4. **Custom Storage Solutions**:
    - The system allows for implementation of custom storage options to meet specific needs.
 
 ## Choosing the Right Storage Option
 
 - Use In-Memory Storage for development, testing, or when persistence between application restarts is not necessary.
 - Choose DynamoDB Storage for production environments where conversation history needs to be preserved long-term or across multiple instances of your application.
+- Consider SQL Storage for a balance between simplicity and scalability, supporting both local and remote databases.
 - Implement a custom storage solution if you have specific requirements not met by the provided options.
 
 ## Next Steps
 
 - Learn more about [In-Memory Storage](/multi-agent-orchestrator/storage/in-memory)
 - Explore [DynamoDB Storage](/multi-agent-orchestrator/storage/dynamodb) for persistent storage
+- Explore [SQL Storage](/multi-agent-orchestrator/storage/sql) for persistent storage using SQLite or Turso.
 - Discover how to [implement custom storage solutions](/multi-agent-orchestrator/storage/custom)
 
 By leveraging these storage options, you can ensure that your Multi-Agent Orchestrator System maintains the necessary context for coherent and effective conversations across various use cases and deployment scenarios.

docs/src/content/docs/storage/sql.mdx

@@ -0,0 +1,144 @@
+---
+title: SQL Storage
+description: Using SQL databases (SQLite/Turso) for persistent conversation storage in the Multi-Agent Orchestrator System
+---
+
+SQL storage provides a flexible and reliable solution for storing conversation history in the Multi-Agent Orchestrator System. This implementation supports both local SQLite databases and remote Turso databases, making it suitable for various deployment scenarios from development to production.
+
+## Features
+
+- Persistent storage across application restarts
+- Support for both local and remote databases
+- Built-in connection pooling and retry mechanisms
+- Compatible with edge and serverless deployments
+- Transaction support for data consistency
+- Efficient indexing for quick data retrieval
+
+## When to Use SQL Storage
+
+- When you need a balance between simplicity and scalability
+- For applications requiring persistent storage without complex infrastructure
+- In both development and production environments
+- When working with edge or serverless deployments
+- When you need local-first development with remote deployment options
+
+## Python Package Installation
+
+To use SQL storage in your Python application, make sure to install them:
+
+```bash
+pip install "multi-agent-orchestrator[sql]"
+```
+
+This will install the `libsql-client` package required for SQL storage functionality.
+
+## Implementation
+
+To use SQL storage in your Multi-Agent Orchestrator:
+
+import { Tabs, TabItem } from '@astrojs/starlight/components';
+
+<Tabs syncKey="runtime">
+  <TabItem label="TypeScript" icon="seti:typescript" color="blue">
+    ```typescript
+    import { SqlChatStorage, MultiAgentOrchestrator } from 'multi-agent-orchestrator';
+
+    // For local SQLite database
+    const localStorage = new SqlChatStorage('file:local.db');
+    await localStorage.waitForInitialization();
+
+    // For remote database
+    const remoteStorage = new SqlChatStorage(
+      'libsql://your-database-url.example.com',
+      'your-auth-token'
+    );
+    await remoteStorage.waitForInitialization();
+
+    const orchestrator = new MultiAgentOrchestrator({
+      storage: localStorage // or remoteStorage
+    });
+
+
+    // Close the database connections when done
+    await localStorage.close();
+    await remoteStorage.close();
+    ```
+  </TabItem>
+  <TabItem label="Python" icon="seti:python">
+    ```python
+    from multi_agent_orchestrator.storage import SqlChatStorage
+    from multi_agent_orchestrator.orchestrator import MultiAgentOrchestrator
+
+    # For local SQLite database
+    local_storage = SqlChatStorage('file:local.db')
+
+    # For remote Turso database
+    remote_storage = SqlChatStorage(
+      url='libsql://your-database-url.turso.io',
+      auth_token='your-auth-token'
+    )
+
+    orchestrator = MultiAgentOrchestrator(storage=local_storage)  # or remote_storage
+    ```
+  </TabItem>
+</Tabs>
+
+## Configuration
+
+### Local DB
+For local development, simply provide a file URL:
+```typescript
+const storage = new SqlChatStorage('file:local.db');
+```
+
+### Remote DB
+For production with Turso:
+1. Create a Turso database through their platform
+2. Obtain your database URL and authentication token
+3. Configure your storage:
+```typescript
+const storage = new SqlChatStorage(
+  'libsql://your-database-url.com',
+  'your-auth-token'
+);
+```
+
+## Database Schema
+
+The SQL storage implementation uses the following schema:
+
+```sql
+CREATE TABLE conversations (
+  user_id TEXT NOT NULL,
+  session_id TEXT NOT NULL,
+  agent_id TEXT NOT NULL,
+  message_index INTEGER NOT NULL,
+  role TEXT NOT NULL,
+  content TEXT NOT NULL,
+  timestamp INTEGER NOT NULL,
+  PRIMARY KEY (user_id, session_id, agent_id, message_index)
+);
+
+CREATE INDEX idx_conversations_lookup 
+ON conversations(user_id, session_id, agent_id);
+```
+
+## Considerations
+
+- Automatic table and index creation on initialization
+- Built-in transaction support for data consistency
+- Efficient query performance through proper indexing
+- Support for message history size limits
+- Automatic JSON serialization/deserialization of message content
+
+## Best Practices
+
+- Use a single instance of SqlChatStorage throughout your application
+- Regularly backup your database
+- Implement data cleanup strategies for old conversations
+- Monitor database size and performance
+- Keep your authentication tokens secure
+- Implement proper access controls at the application level
+- Close the database connections when done
+
+SQL storage provides a robust and flexible solution for managing conversation history in the Multi-Agent Orchestrator System. It offers a good balance between simplicity and features, making it suitable for both development and production environments.

python/setup.cfg

@@ -27,10 +27,13 @@ anthropic =
     anthropic>=0.40.0
 openai =
     openai>=1.55.3
+sql = 
+    libsql-client>=0.3.1
 all =
     anthropic==0.40.0
     openai==1.55.3
     boto3==1.35.0
+    libsql-client==0.3.1
 
 [options.packages.find]
 where = src

python/src/multi_agent_orchestrator/storage/__init__.py

@@ -5,13 +5,20 @@
 from .in_memory_chat_storage import InMemoryChatStorage
 
 _AWS_AVAILABLE = False
+_SQL_AVAILABLE = False
 
 try:
     from .dynamodb_chat_storage import DynamoDbChatStorage
     _AWS_AVAILABLE = True
 except ImportError:
     _AWS_AVAILABLE = False
 
+try:
+    from .sql_chat_storage import SqlChatStorage
+    _SQL_AVAILABLE = True
+except ImportError:
+    _SQL_AVAILABLE = False
+
 __all__ = [
     'ChatStorage',
     'InMemoryChatStorage',
@@ -20,4 +27,9 @@
 if _AWS_AVAILABLE:
     __all__.extend([
         'DynamoDbChatStorage'
+    ])
+
+if _SQL_AVAILABLE:
+    __all__.extend([
+        'SqlChatStorage'
     ])