feat: eval navsim

Author: YoucanBaby

.gitignore

@@ -223,12 +223,11 @@ fusion_result.json
 *output*
 *kernel_meta/
 data/*.json
-maps
 data/metric_cache
-data/maps
 tmp
 navsim_v1
 nuscenes
 extra-info/
 pretrained_model/
-.DS_Store
+.DS_Store
+exp/

README.md

@@ -3,7 +3,7 @@
     <a href='https://github.com/YoucanBaby' target='_blank'>Yifang Xu</a><sup>1*</sup>&emsp;
     <a href='https://cuijh26.github.io/' target='_blank'>Jiahao Cui</a><sup>1*</sup>&emsp;
     <a href='https://github.com/fudan-generative-vision/WAM-Flow' target='_blank'>Feipeng Cai</a><sup>2*</sup>&emsp;
-    <a href='https://github.com/SSSSSSuger' target='_blank'>Zhihao Zhu</a><sup>1</sup>&emsp;
+    <a href='https://github.com/SSSSSSuger' target='_blank'>Zhihao Zhu</a><sup>1*</sup>&emsp;
     <a href='https://github.com/NinoNeumann' target='_blank'>Hanlin Shang</a><sup>1</sup>&emsp;
     <a href='https://github.com/isan089' target='_blank'>Shan Luan</a><sup>1</sup>&emsp;
 </div>
@@ -30,6 +30,7 @@
 
 
 ## 📰 News
+- **`2026/02/21`**: 🎉🎉🎉 WAM-Flow is accepted by [CVPR 2026](https://cvpr.thecvf.com/).
 - **`2026/02/01`**: 🎉🎉🎉 Release the pretrained models on [Huggingface](https://huggingface.co/fudan-generative-ai/WAM-Flow).
 - **`2025/12/06`**: 🎉🎉🎉 Paper submitted on [Arxiv](https://arxiv.org/pdf/2512.06112).
 
@@ -41,7 +42,7 @@
 | :----: | :----------------------------------------------------------------------------------------------------: | :--------: |
 |   ✅   | **[Release the SFT and inference code](https://github.com/fudan-generative-vision/WAM-Flow)**   | 2025.12.19 |
 |   ✅   | **[Pretrained models on Huggingface](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | 2026.02.01        |
-|   🚀   | **[Release the evaluation code](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | TBD |
+|   ✅   | **[Release the evaluation code](https://huggingface.co/fudan-generative-ai/WAM-Flow)**    | 2026.03.03 |
 |   🚀   | **[Release the RL code](https://github.com/fudan-generative-vision/WAM-Flow)**   | TBD |
 |   🚀   | **[Release the pre-processed training data](#training)**                                       | TBD        |
 
@@ -82,13 +83,14 @@ cd WAM-Flow
 Install dependencies:
 
 ```sh
-conda create --name wam-flow python=3.10
+conda create --name wam-flow python=3.9
 conda activate wam-flow
-pip install -r requirements.txt
+pip install -e ./nuplan-devkit
+pip install -e .
 ```
 
 
-### Model Download
+### Model
 
 Download models using huggingface-cli:
 
@@ -99,6 +101,24 @@ huggingface-cli download LucasJinWang/FUDOKI --local-dir ./pretrained_model/fudo
 ```
 
 
+### Dataset
+
+**NAVSIM**
+
+Please [download NAVSIM dataset](https://github.com/autonomousvision/navsim/blob/main/docs/install.md#2-download-the-dataset) and [run metric caching](https://github.com/autonomousvision/navsim/blob/main/docs/cache.md).
+
+
+
+
+### Evaluation
+
+**NAVSIM**
+
+```sh
+# Please change NAVSIM and METRIC_CACHE path
+sh scripts/evaluation/run_wam_flow_agent_pdm_score_evaluation.sh
+```
+
 
 ### Inference
 
@@ -137,4 +157,4 @@ The integration of Vision-Language-Action models into autonomous driving introdu
 
 
 ## 🤗 Acknowledgements
-We gratefully acknowledge the contributors to the [Recogdrive](https://github.com/xiaomi-research/recogdrive), [Janus](https://github.com/deepseek-ai/Janus), [FUDOKI](https://github.com/fudoki-hku/FUDOKI) and [flow_matching](https://github.com/facebookresearch/flow_matching) repositories, whose commitment to open source has provided us with their excellent codebases and pretrained models.
+We gratefully acknowledge the contributors to the [WAM-Diff](https://github.com/fudan-generative-vision/WAM-Diff), [RecogDrive](https://github.com/xiaomi-research/recogdrive), [Janus](https://github.com/deepseek-ai/Janus), [FUDOKI](https://github.com/fudoki-hku/FUDOKI) and [flow_matching](https://github.com/facebookresearch/flow_matching) repositories, whose commitment to open source has provided us with their excellent codebases and pretrained models.

config/pretrain.yaml

@@ -11,7 +11,7 @@ ckpt_path: ""
 train_llm_emb: false
 
 data_list: [
-  "path/to/llava_v1_5_mix665k_2.jsonl"
+  "path/to/llava_v1_5_mix665k_2.jsonl",
   "path/to/dataset_coda_lm.jsonl",
   "path/to/dataset_drivegpt4.jsonl",
   "path/to/dataset_lingoqa.jsonl",

download/download_maps.sh

@@ -0,0 +1,4 @@
+wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-maps-v1.1.zip
+unzip nuplan-maps-v1.1.zip
+rm nuplan-maps-v1.1.zip
+mv nuplan-maps-v1.0 maps

download/download_mini.sh

@@ -0,0 +1,22 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_mini.tgz
+tar -xzf openscene_metadata_mini.tgz
+rm openscene_metadata_mini.tgz
+mv openscene-v1.1/meta_datas mini_navsim_logs
+rm -r openscene-v1.1
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_mini_camera/openscene_sensor_mini_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_mini_camera_${split}.tgz"
+    tar -xzf openscene_sensor_mini_camera_${split}.tgz
+    rm openscene_sensor_mini_camera_${split}.tgz
+done
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_mini_lidar/openscene_sensor_mini_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_mini_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_mini_lidar_${split}.tgz
+    rm openscene_sensor_mini_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/sensor_blobs mini_sensor_blobs
+rm -r openscene-v1.1

download/download_navtrain.sh

@@ -0,0 +1,26 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_trainval.tgz
+tar -xzf openscene_metadata_trainval.tgz
+rm openscene_metadata_trainval.tgz
+mv openscene-v1.1/meta_datas trainval_navsim_logs
+rm -r openscene-v1.1
+
+mkdir -p trainval_sensor_blobs/trainval
+for split in {1..4}; do
+    wget https://s3.eu-central-1.amazonaws.com/avg-projects-2/navsim/navtrain_current_${split}.tgz
+    echo "Extracting file navtrain_current_${split}.tgz"
+    tar -xzf navtrain_current_${split}.tgz
+    rm navtrain_current_${split}.tgz
+
+    rsync -rv current_split_${split}/* trainval_sensor_blobs/trainval
+    rm -r current_split_${split}
+done
+
+for split in {1..4}; do
+    wget https://s3.eu-central-1.amazonaws.com/avg-projects-2/navsim/navtrain_history_${split}.tgz
+    echo "Extracting file navtrain_history_${split}.tgz"
+    tar -xzf navtrain_history_${split}.tgz
+    rm navtrain_history_${split}.tgz
+
+    rsync -rv history_split_${split}/* trainval_sensor_blobs/trainval
+    rm -r history_split_${split}
+done

download/download_private_test_e2e.sh

@@ -0,0 +1,10 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_private_test_e2e.tgz
+tar -xzf openscene_metadata_private_test_e2e.tgz
+rm openscene_metadata_private_test_e2e.tgz
+mv openscene-v1.1/meta_datas/ private_test_e2e_navsim_logs
+rm -r openscene-v1.1
+
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_private_test_e2e.tgz
+tar -xzf openscene_sensor_private_test_e2e.tgz
+rm openscene_sensor_private_test_e2e.tgz
+mv competition_test private_test_e2e_sensor_blobs

download/download_test.sh

@@ -0,0 +1,21 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_test.tgz
+tar -xzf openscene_metadata_test.tgz
+rm openscene_metadata_test.tgz
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_test_camera/openscene_sensor_test_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_test_camera_${split}.tgz"
+    tar -xzf openscene_sensor_test_camera_${split}.tgz
+    rm openscene_sensor_test_camera_${split}.tgz
+done
+
+for split in {0..31}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_test_lidar/openscene_sensor_test_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_test_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_test_lidar_${split}.tgz
+    rm openscene_sensor_test_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/meta_datas test_navsim_logs
+mv openscene-v1.1/sensor_blobs test_sensor_blobs
+rm -r openscene-v1.1

download/download_trainval.sh

@@ -0,0 +1,21 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_metadata_trainval.tgz
+tar -xzf openscene_metadata_trainval.tgz
+rm openscene_metadata_trainval.tgz
+
+for split in {0..199}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_trainval_camera/openscene_sensor_trainval_camera_${split}.tgz
+    echo "Extracting file openscene_sensor_trainval_camera_${split}.tgz"
+    tar -xzf openscene_sensor_trainval_camera_${split}.tgz
+    rm openscene_sensor_trainval_camera_${split}.tgz
+done
+
+for split in {0..199}; do
+    wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/openscene-v1.1/openscene_sensor_trainval_lidar/openscene_sensor_trainval_lidar_${split}.tgz
+    echo "Extracting file openscene_sensor_trainval_lidar_${split}.tgz"
+    tar -xzf openscene_sensor_trainval_lidar_${split}.tgz
+    rm openscene_sensor_trainval_lidar_${split}.tgz
+done
+
+mv openscene-v1.1/meta_datas trainval_navsim_logs
+mv openscene-v1.1/sensor_blobs trainval_sensor_blobs
+rm -r openscene-v1.1

download/download_warmup_two_stage.sh

@@ -0,0 +1,3 @@
+wget https://huggingface.co/datasets/OpenDriveLab/OpenScene/resolve/main/navsim-v2/navsim_v2.1.1_warmup_two_stage.tar.gz
+tar -xzvf navsim_v2.1.1_warmup_two_stage.tar.gz
+rm navsim_v2.1.1_warmup_two_stage.tar.gz