QTRobot-Brain

Status: In active development. Code is experimental, unstable, and mostly undocumented.
Goal: Build a complete, low-resource pipeline for real-time human–machine interaction on the QT robot.

---

Project structure

The project is divided into two main components:

1. Emotion Recognition Module

Building an emotion detection system to make the QT robot more expressive and context-aware.

Related Repositories:

• AffectiveTRM: Previously used for multimodal emotion recognition.

  Note: This repository is not used in the final pipeline as results were inconclusive, but is kept for reference.

Progress:

• Dataset collection
• Model training
• Model testing

---

2. Full Low-Resource Interaction Pipeline

An offline, low-compute pipeline enabling real-time interaction on embedded hardware.

Pipeline components:

• LLM backend (llama.cpp)
• Speech-to-text (Vosk)
• Text-to-speech (Piper)
• Custom agent construction with personalized prompts
• Unified orchestration layer (New "Executor" Thread Architecture)

---

Features Under Development & Roadmap

• Windows & Linux (x64) support using GitHub workflows
• Script tested directly on the QT robot
• New Architecture: Threaded executor with queue-based action management for thread safety.
• Connected Jacket: Support via --JKT flag using QT-Touch.
• API Support: Optional Google Gemini API integration via --API.
• Headless Mode: Run without UI using --no-ui.
• --QT argument in main.py: to use when running on the robot.
• --pytest argument in main.py: used to test the pipeline (excluding the UI).
• --ci-mode argument in download_model.py: replaces the large 8B model with a smaller 350M model.
• --name argument: Define custom robot wake-word.
• Language Switching: Ability to switch the system language between French and English.
• Voice Selection: Option to change the TTS voice gender (Male/Female).
• OpenAI API Integration: Extending --API support to include OpenAI models (currently restricted to Google models).
• User Documentation: Create a comprehensive guide/wiki to facilitate repository usage and setup for new users.

Why make it public?

Building in the open for transparency, feedback, and accountability.
Contributors welcome — feel free to open an issue if you want to help!