Open Virtual Steering - VoiceControl

"Sterzare Necesse Est!" — even when hands are off the handlebars.
This project explores voice-based steering input in virtual cycling environments using TinyML on microcontrollers.

What it means

The phrase is a playful remix of the Latin saying "Navigare Necesse Est" ("To sail is necessary"). Replacing Navigare with Sterzare — the Italian verb for steering — reflects the project’s goal: enhancing interaction through alternative inputs like voice.

• More autonomy: increased user-experience
  • Ride the optimal line through a corner
  • Turn left or right at intersections
  • Tactical positioning in a group

This is part of the broader Open Virtual Steering (OVS) initiative — an open-source collection of research projects focused on steering innovation in virtual cycling.

🗣️ Voice-Driven Steering with TinyML

This repository demonstrates how a voice command recognition model can enable hands-free steering. A trained ML model running on a XIAO ESP32S3 Sense board classifies voice input like:

• "left"   → Turn left
• "noise" → No action
• "right" → Turn right

🎤 Microcontroller: XIAO ESP32S3 plus Sense extension: OV2640 camera sensor, SD-card slot and a digital microphone.
🧠 ML Engine: Edge Impulse (Keyword Spotting)
🪄 BLE Role: Server, implementing a steering profile for virtual cycling platforms

📚 Related OVS Repositories

• BLE-Steering-Server - Bluedroid Host Backend Library
• NimBLE-Steering-Server - NimBLE Host Backend Library
• Open-Virtual-Steering-DiscreteHID – Buttons, Joystick, Rotary Encoder
• Open-Virtual-Steering-MotionIMU – Motion-based input using MPU6050

🧭 How It Works

This firmware serves as a BLE Steering Controller, made up of two integrated layers:

1. 🧱 BLESteeringServer or NimBLESteeringServer
   A modular BLE backend that handles BLE advertisement, characteristic management, and transmission of steering angles to virtual cycling clients.

2. 🎮 Voice-Control HID Handler
   This repository uses the XIAO ESP32S3 Sense and a trained Edge Impulse TinyML model to detect voice commands such as "left", "right", or "noise". Recognized keywords are mapped to steering angles.

The handler performs keyword inference and translation, then in the main loop the corresponding steering value is sent to a BLE-connected client using BLESteeringServer::updateSteeringValue().

Processing pipeline:

1. Voice commands are picked up by the onboard MEMS microphone
2. The trained ML model classifies the command
3. The recognized command is translated to a steering instruction
4. Instruction is transmitted via BLE to the connected client

🧪 Testing & Debugging

Validation was performed using:

• The Edge Impulse debug tools
• Serial logging to verify inference output and BLE messaging
• nRF Connect for Mobile for inspecting BLE behavior and notifications

🧱 Dependencies

• Arduino core for ESP32 (tested to work with esp32 3.2.x)
• XIAO ESP32S3 Board + Sense extension
• KWS Inference Engine generated on XIAO-Arduino model export packed in library ei-xiao-kws-arduino-1.0.9.zip
• This sketch depends on installation of one of the following Steering-Server-Backend-Libraries:

BLE Host	Library	Supported Hardware
NimBLE	NimBLE-Steering-Server	ESP32 (ESP-IDF with NimBLE stack)
Bluedroid	BLE-Steering-Server	ESP32 (ESP-IDF default BLE stack)

These critical libraries encapsulate BLE setup and GATT handling, exposing a unified interface for steering input code.

⚠️ Disclaimer

💡 Research & Independence

This project is not affiliated with, endorsed by, or associated with any commercial virtual cycling platform or steering device manufacturer. It is a research and interoperability initiative designed to explore alternative human interface methods in the context of indoor cycling. All development is conducted independently for educational and experimental purposes.

✅ Compliance & Responsibility

This repository does not include or promote any circumvention of technological protection measures, reverse engineering of proprietary software, or unauthorized access to restricted systems. Users are solely responsible for ensuring that their use of this code complies with local laws, software licenses, and platform terms of service.

🔍 Copyright & Contact

If you are a rights holder and believe this project includes content that violates your intellectual property rights, please open an issue in this repository. We are committed to responding promptly and respectfully to legitimate concerns.

🛠 In Code Preference Selection

In the header of the .ino code file you will find a section, that needs to be aligned with your preferred setup:

// ========================================================
// 🏁 Select BLE Stack
// Uncomment *one* of the following lines to choose the BLE stack
// ========================================================

// #define USE_NIMBLE           // Use NimBLE-Arduino (lightweight)
// #define USE_BLUEDROID        // Use default ESP32 Bluedroid stack

// ========================================================
// 📦 Include appropriate BLE Steering Server library
// ========================================================

#ifdef USE_NIMBLE
  #include <NimBLESteeringServer.h>
#elif defined(USE_BLUEDROID)
  #include <BLESteeringServer.h>
#else
  #error "Please define a BLE stack: USE_NIMBLE or USE_BLUEDROID"
#endif

🚀 Quick Start

1. Download or clone this repository.
2. Unzip (if needed) and move the folder into your Arduino Sketchbook:
3. Open the .ino file manually in Arduino IDE, check Tools settings and set your preferred Backend Library
4. Ensure required libraries are installed (see Dependencies).
5. Compile and upload to your board.

ℹ️ The sketch file name may differ from the repository name. This is intentional for better versioning and description.

🛠️ View recommended Arduino IDE settings for XIAO ESP32S3

Voice Control with XIAO ESP32S3 Sense

Edge Impulse empowers to bring intelligence to embedded projects by enabling devices to understand and respond to their environment. Whether to recognize sounds, identify objects, or detect motion, Edge Impulse makes it accessible and straightforward. Edge Impulse has embraced among other MCU's the XIAO ESP32S3 Sense development board for tinyML projects. Seeed Studio has integrated support for this board into Edge Impulse, allowing users to sample raw data and build machine learning models directly from the studio. This integration simplifies the process of leveraging the XIAO ESP32S3 Sense for tinyML projects.

With its impressive specifications and Edge Impulse compatibility, the XIAO ESP32S3 Sense is an excellent choice for developers seeking to explore embedded machine learning applications.

🔁 TinyML Model Development

Model development followed Edge Impulse’s TinyML KWS workflow:

• Voice data collection
• Labeling and training
• Model optimization for ESP32S3
• Deployment using the Edge Impulse Arduino library
  

All the work has been done and the ready-to-use final result is included in the library: ei-xiao-kws-arduino-1.0.9.zip

🎤 Microphone (MEMS)

The built-in microphone *) is a PDM digital output MEMS microphone with Multi-modes. Internally, it is connected to the ESP32S3 via an I2S bus using ESP32S3 pins IO 42 (IIS_LCLK) and IO 41 (IIS_DOUT).
Notice the two extra leds that have been added to the setup, to indicate successful left/right keyword classification.

*) Microphone type MSM261D3526H1CPM

Google Speech Commands v2

This is an Open Source set of one-second .wav audio files, each containing a single spoken English word. These words are from a small set of commands, and are spoken by a variety of different speakers. The audio files are organized into folders based on the word they contain, and this data set is designed to help train simple machine learning models.
This dataset is covered in more detail at https://arxiv.org/abs/1804.03209. It’s licensed under the Creative Commons BY 4.0 license.
Its original location was at http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz. Version 0.01 of the data set was released on August 3rd 2017 and contained 64,727 audio files. The version 0.02 of the data set (File size: 2.3 GB) is containing 105,829 audio files, and was released on April 11th 2018.

Twenty core command words were recorded, with most speakers saying each of them five times. The core words are “Yes”, “No”, “Up”, “Down”, “Left”, “Right”, “On”, “Off”, “Stop”, “Go”, “Zero”, “One”, “Two”, “Three”, “Four”, “Five”, “Six”, “Seven”, “Eight”, and “Nine”. To help distinguish unrecognized words, there are also ten auxiliary words, which most speakers only said once. These include “Bed”, “Bird”, “Cat”, “Dog”, “Happy”, “House”, “Marvin”, “Sheila”, “Tree”, and “Wow”.

In this project "Left", "Right" and "Noise" recordings were extracted from the file (v0.02), partly renamed and uploaded to "Edge Impulse Studio" to train the model.

Inferencing Engine

The Convolutional Neural Network (CNN) library is built in the (free version of) Edge Impulse Studio and needs to be loaded in the Arduino IDE 2.x. Download Edge Impulse Arduino Deployment file: ei-xiao-kws-arduino-1.0.9.zip and add to the Arduino libraries following the instruction:
Sketch -> Include libray -> Add .Zip Library...
The library is referenced in the code as follows:

#include <XIAO-KWS_inferencing.h>

Neural Network Setup

The Convolutional Neural Network is trained with a set of 6677 items and a test set of 1662 items, in total 2h 17min of 1 sec speech fragments (in wav format).

1D Convolutional Neural Network Architecture, EON Tuned with 6 layers:

• Reshape (32 Columns)
• Pool Layer (16 neurons, 3 kernel size, 1 layer)
• Pool Layer (32, 3, 1)
• Pool Layer (64, 3, 1)
• Flatten Layer
• Dropout (rate 0.5)

Classification Accuracy

The final model accuracy, trained and tested with 2 spoken keywords and a variety of "noise" recordings:

Keyword	Left	Noise	Right
Items	2850	1030	2795
Accuracy	97.2%	95.4%	98.7%

---

⚖️ Legal Notice (EU Context)

This project is developed and published in accordance with EU directives that recognize the right to study, test, and develop software components for the purpose of achieving interoperability (e.g., Directive 2009/24/EC on the legal protection of computer programs, Article 6).

No part of this project is intended to infringe upon intellectual property rights or violate technological protection measures. All content is shared in good faith under the belief that it falls within the bounds of legitimate research, reverse engineering for interoperability, and fair use under EU law.

Users must ensure their own compliance with national implementations of EU directives, and are responsible for how they apply or modify this code.

---

🧠 Contributions, test results, and ML model improvements are very welcome. Let’s make indoor cycling smarter and more inclusive.