Merge pull request #89 from scikit-learn-contrib/joss-feedback

♻️ improve benchmark reproducibility

Author: EmanuelSommer

paper/scripts/README.md

@@ -2,29 +2,42 @@
 
 The scripts in this directory reproduce benchmark results used in the paper.
 
-## Additional dependencies
+## Benchmark environment
 
 The benchmark script (`benchmark.py`) additionally imports:
 
 - `xgboost`
 - `catboost`
 - `tabpfn`
 
-These are **benchmark-only** dependencies. They are intentionally not required by
-the core `sklearn-contrib-bde` package and are therefore not listed under the
-main project dependencies.
+These are **benchmark-only** dependencies. They are intentionally kept out of
+the core `sklearn-contrib-bde` package and provided through the optional pixi
+`benchmark` environment instead.
 
 ## Setup
 
 Run from the repository root unless noted otherwise.
 
-1. Create/install the base project environment as specified in the main project 
-documentation.
+1. Install the benchmark environment:
 
-2. Install benchmark-only Python packages into the pixi environment:
+```bash
+pixi install -e benchmark
+```
+
+This installs the locked benchmark dependencies, including `pip`, `xgboost`,
+`catboost`, and `tabpfn`; no manual `pip install` or `ensurepip` step is needed.
+
+2. Run a benchmark through the pixi task. For example:
 
 ```bash
-pixi run python -m pip install xgboost catboost tabpfn
+pixi run -e benchmark benchmark --dataset airfoil --models linear rf --n-runs 5
 ```
 
-3. Run the benchmark script `paper/scripts/run_bench.sh` to reproduce the benchmark.
+The task invokes `paper/scripts/run_bench.sh`. The wrapper resolves its paths
+relative to its own location, so this equivalent command also works from the
+repository root:
+
+```bash
+pixi run -e benchmark bash paper/scripts/run_bench.sh \
+  --dataset airfoil --models linear rf --n-runs 5
+```

paper/scripts/run_bench.sh

@@ -1,6 +1,10 @@
 #!/usr/bin/env bash
 set -euo pipefail
+
+script_dir="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+
 set -a
-source scripts/bench.env
+source "${script_dir}/bench.env"
 set +a
-python scripts/benchmark.py "$@"
+
+python "${script_dir}/benchmark.py" "$@"