docs: add full train set OL evaluation section and fix rare case paths

Author: WCJ-BERT

docs/algengine_usage.md

@@ -157,6 +157,39 @@ work_dirs/e2e_vadv2_50pct/
 └── navtest_failures.csv        # rare navtest cases only
 ```
 
+#### Full Train Set Evaluation
+
+Evaluate on the full training set (navtrain) to produce per-scenario metrics for [Rare Case Extraction](#rare-case-extraction). Because navtrain is large, the script splits it into chunks to avoid OOM.
+
+```bash
+conda activate algengine
+cd projects/AlgEngine
+
+# Chunked evaluation on navtrain (8 GPUs, 20 chunks)
+bash scripts/e2e_dist_eval_navtrain_chunked.sh \
+    configs/worldengine/e2e_vadv2_50pct.py \
+    work_dirs/e2e_vadv2_50pct/epoch.pth \
+    8 \
+    20
+```
+
+**Arguments:**
+1. `<config>`: Configuration file path
+2. `<checkpoint>`: Model checkpoint to evaluate
+3. `<num_gpus>`: Number of GPUs to use
+4. `[num_chunks]` (optional, default 10): Number of chunks to split navtrain into
+
+The script automatically:
+1. Splits `navtrain.yaml` into chunks under `configs/navsim_splits/navtrain_split/chunks/`
+2. Evaluates each chunk sequentially
+3. Merges all chunk CSVs into a single file
+
+**Output:**
+```
+experiments/worldengine/e2e_vadv2_50pct/
+└── navtrain.csv                # Full train set evaluation results
+```
+
 #### Understanding Evaluation Metrics
 
 Open-loop metrics CSV format:
@@ -204,16 +237,20 @@ bash scripts/run_ray_distributed_testing.sh \
 
 Extract failure scenarios from evaluation results for targeted fine-tuning.
 
+### Prerequisites
+
+Before extracting rare cases, you **must** complete a [Full Train Set Evaluation](#full-train-set-evaluation) to generate `navtrain.csv` with per-scenario metrics. The rare case extraction script uses this CSV to identify failure scenarios.
+
 ### Basic Extraction
 
 ```bash
 conda activate algengine
 cd projects/AlgEngine
 
 python scripts/rare_case_sampling_by_pdms.py \
-    --pdm-result work_dirs/e2e_vadv2_50pct/navtest.csv \
-    --base-split configs/navsim_splits/navtest_split/navtest.yaml \
-    --output-dir configs/navsim_splits/navtest_split/e2e_vadv2_50pct_rare
+    --pdm-result work_dirs/e2e_vadv2_50pct/navtrain.csv \
+    --base-split configs/navsim_splits/navtrain_split/navtrain_50pct.yaml \
+    --output-dir configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare
 ```
 
 **Arguments:**
@@ -264,7 +301,7 @@ Fine-tune a trained model on rare cases using reinforcement learning.
 
 ### Prerequisites: Generating Rollouts with SimEngine
 
-**Important:** Rollout data must be generated by SimEngine before fine-tuning. This involves:
+**Important:** Rare case extraction and Rollout data must be generated by SimEngine before fine-tuning. This involves:
 
 1. **Run SimEngine Rollout** to generate trajectory data:
    ```bash
@@ -343,9 +380,9 @@ data = dict(
     train=dict(
         ann_file='merged_infos_navformer/nuplan_openscene_navtrain.pkl',
         scenario_filter=[
-            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_ep8/navtrain_50pct_collision.yaml',
-            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_ep8/navtrain_50pct_off_road.yaml',
-            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_ep8/navtrain_50pct_ep_1pct.yaml',
+            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_collision.yaml',
+            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_off_road.yaml',
+            'configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_50pct_ep_1pct.yaml',
         ]
     )
 )

docs/simengine_usage.md

@@ -274,8 +274,8 @@ Example:
 conda activate simengine
 
 python projects/SimEngine/worldengine/utils/dataset_utils/nuplan/digitaltwin_nuplan_converter_navsim_filter.py \
-    --navsim-filters $ALGENGINE_ROOT/data_loop/navtrain_split/e2e_hydramdp_ep8/navtrain_collision.yaml \
-        $ALGENGINE_ROOT/data_loop/navtrain_split/e2e_hydramdp_ep8/navtrain_ep_1pct.yaml \
+    --navsim-filters $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_collision.yaml \
+        $ALGENGINE_ROOT/configs/navsim_splits/navtrain_split/e2e_vadv2_50pct_rare/navtrain_ep_1pct.yaml \
     --out-dir data/sim_engine/scenarios/original/navtrain_hydramdp_failures \
     --num-processes 8
 ```