## Kafka Integration
The application now supports comprehensive asynchronous processing using Kafka. This allows for better scalability and performance as system operations are decoupled from the request/response cycle.
### Setting up Kafka
1. Run the Kafka setup script:
```bash
# Linux/MacOS
./server/kafka-setup.sh
# Windows (using WSL or Git Bash)
bash ./server/kafka-setup.sh
```
2. Add the following to your `.env` file:
```
ENABLE_KAFKA=true
KAFKA_BROKERS=localhost:9092
KAFKA_CLIENT_ID=synapse-app
```
3. Restart the application
### Configuration
- `ENABLE_KAFKA`: Set to `true` to enable Kafka processing, `false` to use synchronous processing
- `KAFKA_BROKERS`: Comma-separated list of Kafka brokers (e.g., `localhost:9092,kafka2:9092`)
- `KAFKA_CLIENT_ID`: Client identifier for Kafka connections
### Supported Asynchronous Operations
The application uses Kafka for the following types of operations:
#### Chat Messages
- Processing user messages asynchronously, including AI response generation
- Handling messages from different sources (Web, Telegram, etc.)
#### User Summarization
- Generating and updating user summaries based on interaction history
- Extracting key user traits, interests, and patterns
#### Context Analysis
- Processing user inputs for deeper understanding
- Maintaining companion thinking and state updates
#### Memory Operations
- Creating, updating, and deleting user memories
- Retrieving relevant memory information
#### Session Operations
- Creating and managing user sessions
- Handling session updates and metadata
#### Activity Operations
- Starting, updating, and ending user activities
- Processing activity-related messages
- Updating activity state
#### Action Operations
- Executing user-triggered actions
- Processing action results
### Fallback Mode
If Kafka is unavailable or disabled, the system will automatically fall back to synchronous processing to ensure reliability while sacrificing some scalability.
### Kafka Topics
The system uses multiple Kafka topics for different types of processing:
- `chat-message-requests`: Chat message processing (user to AI interactions)
- `summarization-requests`: User summary generation (session and user analysis)
- `context-analysis-requests`: Context and psychology analysis
### Additional Features
In addition to chat message processing, the following CPU-intensive operations are also offloaded to Kafka:
1. **User Summarization**: The system generates and updates user summaries asynchronously, improving response times while maintaining context.
2. **Context Analysis**: Psychological analysis and AI meta-thinking is processed in the background, allowing the chat interface to remain responsive while deeper analysis happens behind the scenes.
These operations run independently of the main chat flow, allowing the system to scale better under load by processing intensive operations separately from user interactions.