Real-Time AI Voice Chat 🎤💬🧠🔊

Have a natural, spoken conversation with an AI!

This project lets you chat with a Large Language Model (LLM) using just your voice, receiving spoken responses in near real-time. Think of it as your own digital conversation partner.

FastVoiceTalk_compressed_step3_h264.mp4

(early preview - first reasonably stable version)

What's Under the Hood?

A sophisticated client-server system built for low-latency interaction:

1. 🎙️ Capture: Your voice is captured by your browser.
2. ➡️ Stream: Audio chunks are whisked away via WebSockets to a Python backend.
3. ✍️ Transcribe: RealtimeSTT rapidly converts your speech to text.
4. 🤔 Think: The text is sent to an LLM (like Ollama or OpenAI) for processing.
5. 🗣️ Synthesize: The AI's text response is turned back into speech using RealtimeTTS.
6. ⬅️ Return: The generated audio is streamed back to your browser for playback.
7. 🔄 Interrupt: Jump in anytime! The system handles interruptions gracefully.

Key Features ✨

• Fluid Conversation: Speak and listen, just like a real chat.
• Real-Time Feedback: See partial transcriptions and AI responses as they happen.
• Low Latency Focus: Optimized architecture using audio chunk streaming.
• Smart Turn-Taking: Dynamic silence detection (turndetect.py) adapts to the conversation pace.
• Flexible AI Brains: Pluggable LLM backends (Ollama default, OpenAI support via llm_module.py).
• Customizable Voices: Choose from different Text-to-Speech engines (Kokoro, Coqui, Orpheus via audio_module.py).
• Web Interface: Clean and simple UI using Vanilla JS and the Web Audio API.
• Dockerized Deployment: Recommended setup using Docker Compose for easier dependency management.

Technology Stack 🛠️

• Backend: Python 3.x, FastAPI
• Frontend: HTML, CSS, JavaScript (Vanilla JS, Web Audio API, AudioWorklets)
• Communication: WebSockets
• Containerization: Docker, Docker Compose
• Core AI/ML Libraries:
  • RealtimeSTT (Speech-to-Text)
  • RealtimeTTS (Text-to-Speech)
  • transformers (Turn detection, Tokenization)
  • torch / torchaudio (ML Framework)
  • ollama / openai (LLM Clients)
• Audio Processing: numpy, scipy

Before You Dive In: Prerequisites 🏊‍♀️

This project leverages powerful AI models, which have some requirements:

• Operating System:
  • Docker: Linux is recommended for the best GPU integration with Docker.
  • Manual: The provided script (install.bat) is for Windows. Manual steps are possible on Linux/macOS but may require more troubleshooting (especially for DeepSpeed).
• 🐍 Python: 3.9 or higher (if setting up manually).
• 🚀 GPU: A powerful CUDA-enabled NVIDIA GPU is highly recommended, especially for faster STT (Whisper) and TTS (Coqui). Performance on CPU-only or weaker GPUs will be significantly slower.
  • The setup assumes CUDA 12.1. Adjust PyTorch installation if you have a different CUDA version.
  • Docker (Linux): Requires NVIDIA Container Toolkit.
• 🐳 Docker (Optional but Recommended): Docker Engine and Docker Compose v2+ for the containerized setup.
• 🧠 Ollama (Optional): If using the Ollama backend without Docker, install it separately and pull your desired models. The Docker setup includes an Ollama service.
• 🔑 OpenAI API Key (Optional): If using the OpenAI backend, set the OPENAI_API_KEY environment variable (e.g., in a .env file or passed to Docker).

---

Getting Started: Installation & Setup ⚙️

Clone the repository first:

git clone https://github.com/KoljaB/RealtimeVoiceChat.git
cd RealtimeVoiceChat

Now, choose your adventure:

🚀 Option A: Docker Installation (Recommended for Linux/GPU)

This is the most straightforward method, bundling the application, dependencies, and even Ollama into manageable containers.

1. Build the Docker images: (This takes time! It downloads base images, installs Python/ML dependencies, and pre-downloads the default STT model.)

   docker compose build

   (If you want to customize models/settings in code/*.py, do it before this step!)

2. Start the services (App & Ollama): (Runs containers in the background. GPU access is configured in docker-compose.yml.)

   docker compose up -d

   Give them a minute to initialize.

3. (Crucial!) Pull your desired Ollama Model: (This is done after startup to keep the main app image smaller and allow model changes without rebuilding. Execute this command to pull the default model into the running Ollama container.)

   # Pull the default model (adjust if you configured a different one in server.py)
   docker compose exec ollama ollama pull hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M
   
   # (Optional) Verify the model is available
   docker compose exec ollama ollama list

4. Stopping the Services:

   docker compose down

5. Restarting:

   docker compose up -d

6. Viewing Logs / Debugging:

   • Follow app logs: docker compose logs -f app
   • Follow Ollama logs: docker compose logs -f ollama
   • Save logs to file: docker compose logs app > app_logs.txt

🛠️ Option B: Manual Installation (Windows Script / venv)

This method requires managing the Python environment yourself. It offers more direct control but can be trickier, especially regarding ML dependencies.

B1) Using the Windows Install Script:

1. Ensure you meet the prerequisites (Python, potentially CUDA drivers).
2. Run the script. It attempts to create a venv, install PyTorch for CUDA 12.1, a compatible DeepSpeed wheel, and other requirements.

   install.bat

   (This opens a new command prompt within the activated virtual environment.) Proceed to the "Running the Application" section.

B2) Manual Steps (Linux/macOS/Windows):

1. Create & Activate Virtual Environment:

   python -m venv venv
   # Linux/macOS:
   source venv/bin/activate
   # Windows:
   .\venv\Scripts\activate

2. Upgrade Pip:

   python -m pip install --upgrade pip

3. Navigate to Code Directory:

   cd code

4. Install PyTorch (Crucial Step - Match Your Hardware!):

   • With NVIDIA GPU (CUDA 12.1 Example):

     # Verify your CUDA version! Adjust 'cu121' and the URL if needed.
     pip install torch==2.5.1+cu121 torchaudio==2.5.1+cu121 torchvision --index-url https://download.pytorch.org/whl/cu121

   • CPU Only (Expect Slow Performance):

     # pip install torch torchaudio torchvision

   • Find other PyTorch versions: https://pytorch.org/get-started/previous-versions/

5. Install Other Requirements:

   pip install -r requirements.txt

   • Note on DeepSpeed: The requirements.txt may include DeepSpeed. Installation can be complex, especially on Windows. The install.bat tries a precompiled wheel. If manual installation fails, you might need to build it from source or consult resources like deepspeedpatcher (use at your own risk). Coqui TTS performance benefits most from DeepSpeed.

---

Running the Application ▶️

If using Docker: Your application is already running via docker compose up -d! Check logs using docker compose logs -f app.

If using Manual/Script Installation:

1. Activate your virtual environment (if not already active):

   # Linux/macOS: source ../venv/bin/activate
   # Windows: ..\venv\Scripts\activate

2. Navigate to the code directory (if not already there):

   cd code

3. Start the FastAPI server:

   python server.py

Accessing the Client (Both Methods):

1. Open your web browser to http://localhost:8000 (or your server's IP if running remotely/in Docker on another machine).
2. Grant microphone permissions when prompted.
3. Click "Start" to begin chatting! Use "Stop" to end and "Reset" to clear the conversation.

---

Configuration Deep Dive 🔧

Want to tweak the AI's voice, brain, or how it listens? Modify the Python files in the code/ directory.

⚠️ Important Docker Note: If using Docker, make any configuration changes before running docker compose build to ensure they are included in the image.

• TTS Engine & Voice (server.py, audio_module.py):

  • Change START_ENGINE in server.py to "coqui", "kokoro", or "orpheus".
  • Adjust engine-specific settings (e.g., voice model path for Coqui, speaker ID for Orpheus, speed) within AudioProcessor.__init__ in audio_module.py.

• LLM Backend & Model (server.py, llm_module.py):

  • Set LLM_START_PROVIDER ("ollama" or "openai") and LLM_START_MODEL (e.g., "hf.co/..." for Ollama, model name for OpenAI) in server.py. Remember to pull the Ollama model if using Docker (see Installation Step A3).
  • Customize the AI's personality by editing system_prompt.txt.

• STT Settings (transcribe.py):

  • Modify DEFAULT_RECORDER_CONFIG to change the Whisper model (model), language (language), silence thresholds (silence_limit_seconds), etc. The default base.en model is pre-downloaded during the Docker build.

• Turn Detection Sensitivity (turndetect.py):

  • Adjust pause duration constants within the TurnDetector.update_settings method.

• SSL/HTTPS (server.py):

  • Set USE_SSL = True and provide paths to your certificate (SSL_CERT_PATH) and key (SSL_KEY_PATH) files.
  • Docker Users: You'll need to adjust docker-compose.yml to map the SSL port (e.g., 443) and potentially mount your certificate files as volumes.
  Generating Local SSL Certificates (Windows Example w/ mkcert)

  1. Install Chocolatey package manager if you haven't already.
  2. Install mkcert: choco install mkcert
  3. Run Command Prompt as Administrator.
  4. Install a local Certificate Authority: mkcert -install
  5. Generate certs (replace your.local.ip): mkcert localhost 127.0.0.1 ::1 your.local.ip
     • This creates .pem files (e.g., localhost+3.pem and localhost+3-key.pem) in the current directory. Update SSL_CERT_PATH and SSL_KEY_PATH in server.py accordingly. Remember to potentially mount these into your Docker container.

---

Contributing 🤝

Got ideas or found a bug? Contributions are welcome! Feel free to open issues or submit pull requests.

License 📜

The core codebase of this project is released under the MIT License (see the LICENSE file for details).

This project relies on external specific TTS engines (like Coqui XTTSv2) and LLM providers which have their own licensing terms. Please ensure you comply with the licenses of all components you use.