HCockpit

Overview

HCockpit is a sophisticated agent architecture designed to enhance situation awareness in automotive cockpit environments. It integrates advanced Large Multimodal Models (LMMs) to improve communication and collaboration between human drivers and autonomous driving systems.

This project introduces HCopilot, an AI copilot agent based on the HCockpit architecture, and evaluates its performance using the GTAV simulation environment.

For more detailed insights, refer to the final report, presentation slides, and other resources in the docs directory.

Getting Started

Installation

Follow these steps to set up HCockpit on your local machine:

1. Clone the Repository

   git clone https://github.com/Wiederholung/HCockpit.git

2. Navigate to the Project Directory

   cd HCockpit/

3. Set Up the Environment and Dependencies

   • Using Conda (Recommended):

     conda env create -f environment.yaml
     conda activate hcockpit

   • Using Pip:

     # Ensure you are using Python 3.10.14 and have activated a virtual environment named 'hcockpit'
     pip install -r requirements.txt

4. Configure the Environment Variables

   • Modify the file .env.template following the instructions inside, and save it as .env.

   • Alternatively, set environment variables according to the .env.template file.

Usage

• Quick Demo: Run all cells in the notebook to see a demo in the change_lane scenario.

• Testing Different Scenarios: Modify the scenario variable in the Load Data cell of the notebook:

  scenarios = ["change_lane", "turn_right-r", "turn_right-l", "dead_zone", "dazzle", "phone"]
  scenario = scenarios[n]  # Change 'n' to select the scenario

  After setting the scenario, execute the Load Data and Main cells again.

• Customizing Prompts: To test different prompts, edit the prompt markdown file and rerun the relevant notebook cells starting from Orchestrate Prompt.

For further details on how to utilize HCockpit, consult the notebook guide.