submit benchmask

Author: aeiwz

.github/workflows/benchmark.yml

@@ -0,0 +1,106 @@
+name: Cross-OS Performance Benchmark
+
+# Runs scripts/perf_report.py on Linux, Windows, and macOS GitHub-hosted
+# runners, building the native C extension from source on each. This produces
+# the multi-platform, OpenMP-enabled numbers the laptop benchmark could not,
+# and uploads them as downloadable artifacts.
+#
+# Trigger manually: Actions tab -> "Cross-OS Performance Benchmark" -> Run workflow.
+# Note: GitHub-hosted runners have NO GPU, so the CUDA path is still not exercised.
+
+on:
+  workflow_dispatch:
+    inputs:
+      quick:
+        description: "Quick mode (small datasets, faster)"
+        type: boolean
+        default: false
+      reps:
+        description: "Repetitions per benchmark (0 = perf_report default of 5)"
+        type: string
+        default: "0"
+  push:
+    tags:
+      - "bench-*"        # push a tag like bench-v9.0.0 to run automatically
+
+concurrency:
+  group: benchmark-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  benchmark:
+    name: benchmark (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Enable OpenMP toolchain (macOS)
+        if: runner.os == 'macOS'
+        run: |
+          brew install libomp
+          PREFIX="$(brew --prefix libomp)"
+          # setup.py's OpenMP probe and the linker need these paths
+          echo "CPATH=$PREFIX/include" >> "$GITHUB_ENV"
+          echo "LIBRARY_PATH=$PREFIX/lib" >> "$GITHUB_ENV"
+          echo "DYLD_LIBRARY_PATH=$PREFIX/lib" >> "$GITHUB_ENV"
+
+      - name: Install dependencies
+        shell: bash
+        run: |
+          python -m pip install --upgrade pip setuptools wheel
+          python -m pip install numpy scipy scikit-learn pandas plotly
+
+      - name: Build metbit from source (compiles native + OpenMP extension)
+        shell: bash
+        run: |
+          python -m pip install -e . --no-build-isolation --force-reinstall
+
+      - name: Report active backend
+        shell: bash
+        run: |
+          python - <<'PY'
+          from metbit._native import backend_info
+          info = backend_info()
+          for k, v in info.items():
+              print(f"{k} = {v}")
+          if not info["native_c"]:
+              print("::warning::native C extension NOT active on this runner - "
+                    "benchmark measures only the NumPy fallback.")
+          PY
+
+      - name: Run benchmark (OpenMP thread sweep)
+        shell: bash
+        run: |
+          set -e
+          OS="${{ matrix.os }}"
+          EXTRA=""
+          [ "${{ inputs.quick }}" = "true" ] && EXTRA="$EXTRA --quick"
+          [ "${{ inputs.reps }}" != "0" ] && [ -n "${{ inputs.reps }}" ] && EXTRA="$EXTRA --reps ${{ inputs.reps }}"
+          CORES="$(python -c 'import os;print(os.cpu_count() or 1)')"
+          THREADS="$(printf '1\n%s\n' "$CORES" | sort -un)"
+          echo "Runner cores: $CORES ; sweeping OMP threads: $(echo $THREADS)"
+          for T in $THREADS; do
+            echo "===== OMP_NUM_THREADS=$T ====="
+            OMP_NUM_THREADS="$T" METBIT_N_JOBS="$T" \
+              python scripts/perf_report.py --output "reports/ci/${OS}/omp_${T}" $EXTRA
+          done
+
+      - name: Upload results
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-${{ matrix.os }}
+          path: reports/ci/**
+          if-no-files-found: error
+          retention-days: 90

.gitignore

@@ -56,3 +56,9 @@ ENV/
 manuscript/incob2026/~$cob2026_manuscript.docx
 /manuscript
 /manuscript
+
+# Benchmark large intermediate output (regenerate via benchmark_mtbls1.py)
+/Benchmark/MTBLS1/result/spectral_matrix_normalised.csv
+
+# Benchmark raw datasets (large; fetch per each dataset's PROVENANCE/README)
+/Benchmark/data/

Benchmark/MTBLS1/README.md

@@ -0,0 +1,52 @@
+# metbit MTBLS1 Full-Pipeline Benchmark
+
+Reproduces the full metbit workflow on the public MTBLS1 human urine <sup>1</sup>H NMR
+study (Salek et al., 2007; via MetaboLights), from raw Bruker FID archives
+through preprocessing, alignment, normalisation, PCA, OPLS-DA, VIP scoring,
+and STOCSY. Referenced from the manuscript's Data Availability statement.
+
+## Scope
+
+- **Performance** — wall-clock time and peak memory (via `tracemalloc`) for
+  each pipeline stage.
+- **Reproducibility** — same-seed determinism check: the stochastic stages
+  (OPLS-DA cross-validation) are re-fit `n=3` times against the same
+  preprocessed matrix with a fixed `random_state`, comparing R2Y, Q2, and the
+  top-10 VIP-ranked ppm positions across runs.
+- **Usability** — the pipeline is a single Python function; the report
+  records its length and the number of distinct metbit API calls needed to
+  go from raw FID directories to a STOCSY connectivity plot, with no manual
+  intervention between stages.
+
+## Running
+
+```bash
+python Benchmark/MTBLS1/benchmark_mtbls1.py
+```
+
+Requires the raw MTBLS1 FID archive (132 Bruker directories) at
+`manuscript/benchmark/data/MTBLS1/FILES` and sample metadata at
+`manuscript/benchmark/data/MTBLS1/samples_parsed.csv`. These are fetched from
+MetaboLights separately (not part of this script, to avoid re-downloading
+~132 FID directories on every run) and are gitignored due to size.
+
+## Output (`result/`)
+
+| File | Contents |
+|---|---|
+| `REPORT.md` | Combined human-readable summary of all three scopes |
+| `performance_report.json` | Per-stage timing, peak memory, environment info |
+| `reproducibility_report.json` | Same-seed determinism metrics |
+| `usability_report.json` | API-call / lines-of-code usability metrics |
+| `opls_metrics.json` | R2Y, Q2, groups, top-VIP STOCSY anchor |
+| `spectral_matrix_normalised.csv` | PQN-normalised, aligned spectral matrix (float32) |
+| `pca_scores.csv` | PCA scores (PC1-PC5) per sample |
+| `vip_scores.csv` | VIP score per spectral feature |
+
+## Caveats
+
+Disease status in MTBLS1 is completely confounded with acquisition batch
+(see manuscript Section 2.13), and cross-validation here is performed at the
+spectrum level, not the participant level. R2Y/Q2/VIP values in this
+benchmark are workflow-execution outputs, not evidence of disease
+discrimination — consistent with how they are reported in the manuscript.