Skip to content

README.md

Latest commit

 

History

History
286 lines (202 loc) · 9.81 KB

README.md

File metadata and controls

286 lines (202 loc) · 9.81 KB
Raw

UCSD ECEMAE 148 Team 14 Final Project — Language Controlled Robot

Language Controlled Robot

ECEMAE 148 Final Project — Team 14, Spring 2026

Table of Contents

  1. Team Members

  2. Final Project

  3. System Overview

  4. Robot Design

  5. How to Run

  6. Known Issues and Improvements

  7. Lessons Learned

  8. Acknowledgments

  9. Authors and Contact

Team Members

Name Major / Role
Aaron Aviles Mechanical Engineering
Makaio Richman Mechanical Engineering
Rohan Nambimadom Electrical and Computer Engineering
Ole Hennig UPS / Project Contributor

Final Project

Project Goal

The goal of this project was to develop a language-controlled UCSD RoboCar that can receive spoken commands, interpret those commands locally, and execute vehicle motion through the ROS 2 control stack. The project also explored vision-based target following using the OAK-D camera.

The system combines speech recognition, local language interpretation, ROS 2 communication, VESC motor control, and OAK-D camera perception to create a more natural interface for controlling the robot.

Original Goals

Must Have

  • Voice commands for vehicle direction and speed control
  • Basic commands such as forward, stop, left, right, faster, slower, and reverse
  • ROS 2 integration with the existing UCSD RoboCar stack
  • Command execution through the VESC motor controller
  • LED feedback to confirm that a command was received

Nice to Have

  • Complex or unstructured voice commands
  • Bilingual command recognition
  • Vision tracking using the OAK-D camera
  • Person or color-based following behavior

Goals We Met

  • Implemented voice command control for key driving actions
  • Added English and German speech recognition using constrained Vosk models
  • Used a local Qwen command interpretation model to map text input to robot commands
  • Sent interpreted commands through the ROS 2 bridge to the VESC controller
  • Added speed-related commands such as faster, slower, and reverse
  • Created a proof-of-concept vision tracking system using the OAK-D camera
  • Used HSV color filtering to detect red target regions
  • Used stereo depth to estimate target distance and publish target position, range, and confidence

If We Had Another Week

  • Add LED feedback for robot status and command confirmation
  • Use LED color or brightness to indicate current speed or command state
  • Improve speech recognition in noisy environments
  • Add noise filtering for the microphone input
  • Use a wireless microphone to reduce motor-noise interference
  • Replace HSV-based color tracking with a YOLO-based human detection model
  • Improve vision tracking so the robot is less likely to confuse the target with similar colors in the background
  • Test the full system more extensively in different lighting and floor conditions

System Overview

The system is separated into four main stages:

  1. Hardware Inputs

    • USB microphone for voice commands
    • OAK-D Lite camera for vision tracking

  2. Perception and Intent

    • Vosk speech recognition converts spoken commands into text
    • Local Qwen model interprets the text command
    • OAK-D camera provides RGB and stereo depth data
    • HSV filtering detects red target regions for tracking

  3. Decision and Control

    • The interpreted command is converted into a motion command
    • Vision target information is used for following behavior
    • Speed and steering values are selected based on the current command or target

  4. Actuation

    • ROS 2 publishes drive commands
    • VESC controls the motor and steering servo
    • The robot executes the desired motion
flowchart LR
    A[USB Microphone] --> B[Vosk Speech Recognition]
    B --> C[Local Qwen Command Interpreter]
    C --> D[ROS 2 Command Bridge]
    D --> E[VESC Controller]
    E --> F[Motor and Steering Servo]
    F --> G[Robot Motion]

    H[OAK-D Lite Camera] --> I[RGB + Stereo Depth]
    I --> J[HSV Target Detection]
    J --> K[Target Position, Range, Confidence]
    K --> D
Loading

Robot Design

Hardware

Component Purpose
Raspberry Pi 5 Main onboard computer
OAK-D Lite Camera RGB vision and stereo depth
USB Microphone Voice command input
VESC Motor Controller Motor and steering control
RC Car Chassis Physical drive platform
Battery System Power for robot electronics and drive system

Software Architecture

The project uses ROS 2 nodes inside the UCSD RoboCar Docker environment. The main software pipeline converts microphone input into text, interprets that text as a driving command, and sends the command through ROS 2 to control the VESC. A separate vision path uses the OAK-D camera for red target tracking.

The language-control path and vision-tracking path both connect to the robot control system, allowing the car to respond to spoken commands and perform basic target-following behavior.

Main ROS 2 Nodes

Node / Package Role
ucsd_robocar_sensor2_pkg Handles sensor input such as camera or microphone-related data
ucsd_robocar_vision2_pkg Processes OAK-D camera frames and vision tracking
ucsd_robocar_control2_pkg Converts interpreted commands into control behavior
ucsd_robocar_actuator2_pkg Sends steering and throttle commands to the VESC
vesc_twist_node / VESC interface Connects ROS 2 commands to the motor controller

How to Run

Step 1 — Enter the Docker Container

docker start t14robocar
docker exec -it t14robocar bash

Step 2 — Source ROS 2

source /opt/ros/jazzy/setup.bash
source /home/projects/ros2_ws/install/setup.bash

If helper commands are configured:

source_ros2

Step 3 — Build the Workspace

cd /home/projects/ros2_ws
colcon build
source install/setup.bash

Or, if helper commands are configured:

build_ros2
source_ros2

Step 4 — Run the Project

Use the launch file or node command for the current version of the project. Example:

ros2 launch ucsd_robocar_vision2_pkg voice_vision_bringup.launch.py

Or run individual nodes for testing:

ros2 run ucsd_robocar_control2_pkg voice_bridge
ros2 run ucsd_robocar_vision2_pkg oakd_camera_node
ros2 run ucsd_robocar_vision2_pkg red_shirt_target_node
ros2 run ucsd_robocar_vision2_pkg person_follow_controller
ros2 run ucsd_robocar_vision2_pkg intent_safety_mux

Step 5 — Test Commands

Example voice commands include:

  • “Forward”
  • “Stop”
  • “Left”
  • “Right”
  • “Faster”
  • “Slower”
  • “Reverse”

For German control, use the supported German command set configured in the Vosk model and command interpreter.

Known Issues and Improvements

Issue Possible Improvement
Latency still affects responsiveness Use a Jetson Nano or other stronger onboard compute platform
Microphone is close to motor noise Use a wireless microphone or external microphone placement
Speech recognition is not always accurate Add noise filtering and better command constraints
German Vosk model can be accent-dependent Test with more speakers and tune command phrases
HSV vision tracking can confuse similar colors Replace HSV tracking with a YOLO-based person detector
LED feedback was not completed Add LED status indicators for command received, ready state, and tracking state

Lessons Learned

  • Project scope can change quickly as new features are added.
  • Voice control and vision tracking both require careful tuning to work reliably on a moving robot.
  • Local command interpretation can reduce dependence on cloud services but still creates latency tradeoffs.
  • Constrained commands are more reliable than fully open-ended language commands.
  • HSV tracking is useful for a proof of concept, but a trained object detection model would be more robust.
  • Hardware placement matters, especially for microphones near motors and electronics.

Acknowledgments

Thank you to Professor Jack Silberman, the ECEMAE 148 teaching team, and the UCSD RoboCar course staff for providing the robot platform, Docker environment, and project support.

Authors and Contact

Name Contact
Aaron Aviles aaaviles@ucsd.edu
Makaio Richman mmrichman@ucsd.edu
Rohan Nambimadom rnambimadom@ucsd.edu
Ole Hennig ole.hennig@tum.de