VisionPilot drive-by-wire bridge with OpenPilot

[m-zain-khawaja]

Description

VisionPilot is designed to be an production-grade L2 ADAS stack used by automotive OEMs and Tier-1 suppliers. In a production setting, we can work with the OEM/Tier-1 to integrate the outputs of VisionPilot with the OEM's drive-by-wire interface.

However, for testing purposes during the development phases of the system, we need to interface VisionPilot with off-the-shelf drive-by-wire providers that can be integrated into stock vehicles. One such drive-by-wire provider is the open-source project by Comma.ai which supplies a vehicle harness and software libraries to support drive-by-wire integration.

Read data from CAN Bus

Create a CAN bus reader class that can leverage any off the shelf OBD port reader device which plugs into the OBD port of the car to read the car's current speed and current steering angle

Data input to Vision Pilot from OBD port

• current speed
• current steering angle

Drive by wire output from Vision Pilot to vehicle

Create a DBW class that interfaces with the Comma.AI harness to send appropriate control signals to control the vehicle's driving and investigate what these signals should be, there are multiple potential options

Data output from VisionPilot to ADAS ECU in vehicle

• (lateral) : steering angle, curvature, lateral acceleration etc
• (longitudinal): acceleration, speed, etc.

Desired Result

Any user with a car which is supported by Comma AI's drive-by-wire kit should be able to use a standalone computer, standalone camera, and harness kit from Comma AI to enable closed-loop L2 ADAS testing using Vision Pilot:

• Standalone computer : runs VisionPilot stack
• Standalone camera : captures video and sends it to Standalone computer
• Comma AI harness: capture CAN bus vehicle state and sends it to Standalone computer & read desired control commands from VisionPilot and sends it to the vehicle's stock ADAS ECU for closed loop control

[tranchithanh2000]

Hello @m-zain-khawaja , after a few days investigated I would like to propose the following implementation plan for this task

1. Requirement Analysis

Functional Requirements

• Inbound: Read and decode CAN signals (e.g., Speed, Angle) via Panda Repo in real-time from the car’s CAN bus.
• Processing: Vision Pilot takes vehicle state and plans the path (Trajectory) using car data + camera.
• Outbound: Convert path commands into CAN messages (e.g., speed, angle) and send to car.
• Safety: Must Override (disconnect) immediately if the driver steers or brakes.

Non-Functional Requirements

• Latency: The delay from "seeing" a frame to "sending" a command must be extremely low.
• Reliability: The system must be able to handle CAN errors or corrupted video frames.
• Portability: The DBW class should be easily configurate for different vehicle models based on the opendbc.

2. System Design

The system is split into 3 main layers:

• Communication Layer: Use the Panda hardware communicates with the car.
  + Input: Raw data from the car's OBD-II Car Harness.
  + Output: Commands (Raw CAN Frames) sent back to the car.

• Interface Layer:

  • CANReader Class: Uses opendbc to map CAN IDs to variables (Speed, Angle, etc.). Provides this data to VisionPilot via a Pub/Sub system (e.g., ROS2 Topic or Shared Memory).
  • CommaDBW Class: Receives commands from VisionPilot. Converts these parameters into specific CAN messages (e.g., speed, angle) based on the manufacturer’s protocol in opendbc.

• Application Layer : Vision Pilot Stack runs the AI models and decides where to drive, while the CANReader provides the necessary feedback to maintain a closed-loop control system.

3. Task Planning

• Phase 1: Implement Inbound Interface

• Hardware Connection & Raw Capture: Connect the Panda to PC and read raw data applying for a supported vehicle in the opendbc.

• Signal Decoding: Use opendbc to turn that raw data into a live (e.g., speed, angle) display. Use Cabana for visualizing CAN frames.

• Latency & Accuracy Validation: Check for lag. Make sure the data on PC matches the car’s dashboard instantly.

• Phase 2: Implement Outbound Interface

• Command Mapping: Write the math logic to turn Vision Pilot's data into an output CAN command.

• Translation to Actuators: Translate the normalized values into signals of hardware

• Phase 3: Integration To Vision Pilot Pipeline
  TBD.

• Phase 4: Testing and Validation
  TBD.

Reference:

Panda: CAN to USB bridge, link PC to car

• Hardware: https://comma.ai/shop/panda
• Github: https://github.com/commaai/panda

Car Harness: A plug-and-play adapter to intercept car signals

• Hardware: https://comma.ai/shop/car-harness

Opendbc: Database converts raw car data into readable units (e.g., speed, angle).

• Github: https://github.com/commaai/opendbc

Cabana (CAN debugging): Visualizer tool used to monitor and decode CAN signals in real-time

• Github: https://github.com/commaai/cabana

[m-zain-khawaja]

@tranchithanh2000 - looks great 👍

[rasarav]

Helping to triage this issue

[m-zain-khawaja]

Dunlevon Team

• Investigate Tata Harrier camera harness and CAN signals to assess feasibility of DBW integration with Comma AI - useful resource. We can use any OBD2->USB reader to read CAN messages from the vehicle and capture the vehicle state, e.g. steering angle, steering torque, accelerator pedal percentage, etc.

High Level Diagram

[tranchithanh2000]

Hello @m-zain-khawaja and @rasarav , after investigating the system requirements over the past few days, I would like to propose the following template. I would greatly appreciate any feedback or suggestions.

Goal:

I propose this drive by wire (DBW) interface template to integrate the VisionPilot stack with any vehicle supported by the Comma AI ecosystem.
Supported Vehicle: https://github.com/commaai/openpilot/blob/master/docs/CARS.md

Detailed designs:

1. Implement Inboud and outboud classes

• [Inbound] DBW_CANReader.py: read CAN from vehicle via Panda, that translates raw vehicle state into usable data for the Vision stack.

[Vehicle CAN Bus]
    ↓ (Steering angle, Speed)
[Panda Hardware - STM32H725]
    ├─ CAN transceiver (3 buses)
    ├─ Message filtering
    └─ Safety validation
    ↓ (USB bulk transfer)
[CANReader Class]
    ├─ Unpack CAN frames
    ├─ Parse with DBC definitions (opendbc)
    └─ Extract signal values
    ↓
[Signal Processing]
    ├─ Steering angle (degrees)
    └─ Current speed (m/s or mph)
    ↓
[Vision Pilot Application]

• [Outbound] DBW_Controller.py: send data from Vision Pilot to Vehicle, that translates Vision Stack’s decisions into actionable vehicle commands.

[Vision Pilot Application]
    ├─ Desired (lateral) : steering angle, curvature, lateral acceleration
    └─ Desired (longitudinal): acceleration, speed
    ↓
[DBWController Class]
    ├─ Validate commands against safety limits
    ├─ Format as CAN messages
    ├─ Apply DBC signal encoding (opendbc)
    └─ Create command frames
    ↓ (USB bulk transfer)
[Panda Hardware - STM32H725]
    ├─ Safety model validation (per car brand)
    ├─ Rate limiting checks
    ├─ Timeout detection
    └─ Message forwarding
    ↓
Vehicle CAN Bus
    ↓
[Vehicle Stock ADAS ECU]
    ├─ Receives lateral (steering) commands
    ├─ Analyze RX Message (Use Cabana Tool to visualize)
    └─ Executes steering control
    ↓
[Vehicle Actuators]
    ├─ Braking
    └─ Steering motor turns wheel

2. Integrate to Vision Pilot Pipeline

Align with app engineer > TBD. 

3. Testing and Validation:

- Debug and Visualize TX & RX data on Cabana tool
- TBD.	

4. Performance Optimization & Safety Complicene

Performace optimization: TBD.
Safety: https://github.com/commaai/openpilot/blob/master/docs/SAFETY.md

Additional information:

1. Panda Red Block Properties:
   Product link: https://comma.ai/shop/panda
   Baud rate: 500kbps.
   CAN channel:

• CAN0 (Main): Powertrains CAN bus (Speed, Engine, etc).
• CAN1: Radar CAN bus.
• CAN2: Camera CAN bus.

[tranchithanh2000]

@tranchithanh2000 Join slot 2