Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions planning/autoware_planning_data_analyzer/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,12 @@ cmake_minimum_required(VERSION 3.14)
project(autoware_planning_data_analyzer)

find_package(autoware_cmake REQUIRED)
find_package(tf2_geometry_msgs REQUIRED)
autoware_package()

ament_auto_add_library(${PROJECT_NAME} SHARED
src/buffer.cpp
src/base_evaluator.cpp
src/open_loop_evaluator.cpp
src/or_event_extractor.cpp
src/or_scene_evaluator.cpp
src/metrics/deviation_metrics.cpp
src/metrics/epdms_aggregation.cpp
src/metrics/ego_progress.cpp
Expand All @@ -29,7 +26,6 @@ ament_auto_add_library(${PROJECT_NAME} SHARED
src/utils/path_utils.cpp
src/autoware_planning_data_analyzer_node.cpp
)
ament_target_dependencies(${PROJECT_NAME} tf2_geometry_msgs)

rclcpp_components_register_node(${PROJECT_NAME}
PLUGIN "autoware::planning_data_analyzer::AutowarePlanningDataAnalyzerNode"
Expand Down Expand Up @@ -67,5 +63,4 @@ ament_auto_package(
INSTALL_TO_SHARE
config
launch
scripts
)
47 changes: 3 additions & 44 deletions planning/autoware_planning_data_analyzer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@

This package provides offline evaluation tools for trajectory planning performance analysis from recorded rosbag data.

## Evaluation Modes

1. **Open Loop**: Evaluate prediction accuracy against ground truth trajectories
2. **OR Scene**: Evaluate override regression scenarios (LIVE vs HISTORICAL)

## Open-Loop Evaluation

Open-loop evaluation computes both trajectory error metrics and planning-quality subscores.
Expand Down Expand Up @@ -63,31 +58,6 @@ ros2 run autoware_planning_data_analyzer autoware_planning_data_analyzer_node --
-p json_output_path:=~/results.json
```

### OR Scene Evaluation

```sh
# Using the unified script
./install/autoware_planning_data_analyzer/share/autoware_planning_data_analyzer/scripts/run_evaluation.sh \
-b ~/result_bag.mcap \
-m or_scene \
-t /planning/diffusion_planner/trajectory \
-o ~/results \
--live \
--viz
```

### Multi-Model Pipeline

```sh
# Evaluate multiple models from DLR results
./install/autoware_planning_data_analyzer/share/autoware_planning_data_analyzer/scripts/multi_model_pipeline.sh \
--scenario ~/dataset/scenario.yaml \
--trajectory /planning/model_a/trajectory \
--trajectory /planning/model_b/trajectory \
--output ~/comparison \
--viz
```

## Output Files

- **`evaluation_result.json`** - Detailed metrics (ADE, FDE, TTC, etc.) and summary statistics
Expand All @@ -96,28 +66,17 @@ ros2 run autoware_planning_data_analyzer autoware_planning_data_analyzer_node --

## Parameters

Key parameters in `config/offline_evaluation.param.yaml`:
Key parameters in `config/planning_data_analyzer.param.yaml`:

- `evaluation.mode`: Evaluation mode (`open_loop` or `or_scene`)
- `evaluation.mode`: Evaluation mode (`open_loop`)
- `evaluation_interval_ms`: Sampling interval (default: 100ms)
- `sync_tolerance_ms`: Time synchronization tolerance (default: 100ms)
- `trajectory_topic`: Trajectory topic to evaluate
- `or_scene_evaluation.*`: OR scene specific settings

### Utility Scripts

Located in `scripts/` directory:

- **Bag Processing**: `merge_bags.py`, `rename_bag_topic.py`, `add_prefix_to_bag.py`
- **Ground Truth**: `add_gt_trajectory_to_bag.py`
- **OR Analysis**: `detect_or_and_route.py`, `evaluate_or_segments.py`
- **Visualization**: `generate_or_visualization.py`
- **Comparison**: `compare_live_historical.py`
- `open_loop.*`: Open-loop specific settings

---

## For More Information

- Offline evaluation scripts: [scripts/](scripts/)
- Configuration: [config/planning_data_analyzer.param.yaml](config/planning_data_analyzer.param.yaml)
- Launch files: [launch/](launch/)
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/**:
ros__parameters:
# Evaluation Mode
# Options: "open_loop" or "or_scene"
# Options: "open_loop"
evaluation:
mode: "open_loop"

Expand Down Expand Up @@ -60,34 +60,3 @@
lane_keep:
max_lateral_deviation: 0.5 # Maximum absolute lateral deviation allowed for LK [m]
max_continuous_violation_time: 2.0 # Continuous over-threshold duration allowed for LK [s]

# OR Scene Evaluation Configuration (only used when mode="or_scene")
or_scene_evaluation:
# Time window configuration (symmetric around OR event)
time_window_sec: 0.5 # Time window on each side of OR event [OR - 0.5s, OR + 0.5s]

# Trajectory source selection
evaluate_live_trajectories: true # true = evaluate result_bag (DLR output with LIVE trajectories)
# false = evaluate input_bag (HISTORICAL recorded trajectories)
input_bag_path: "" # Path to input bag (only used if evaluate_live_trajectories=false)

# Topics to monitor
control_mode_topic: "/vehicle/status/control_mode" # Topic for OR detection (ControlModeReport)

# OR event extraction (Stage 1)
skip_or_extraction: false # Set true if or_events.json already exists
or_events_input_path: "" # Path to pre-extracted or_events.json (optional)
or_events_output_path: "~/or_events.json" # Where to save extracted OR events

# Success criteria (optional - disabled by default)
success_criteria:
enabled: false # Set true to enforce thresholds
max_ade: 1.0 # Maximum allowed ADE (meters)
max_fde: 1.5 # Maximum allowed FDE (meters)
max_lateral_deviation: 0.5 # Maximum absolute lateral deviation in the vehicle frame (meters)
min_ttc: 3.0 # Minimum required TTC (seconds)

# Debug visualization
enable_debug_visualization: false # Generate debug images showing predicted vs GT trajectories
debug_output_dir: "~/or_scene_debug_images" # Directory for debug visualization images
map_path: "" # Optional: path to lanelet2_map.osm for lane visualization
Loading
Loading