Skip to content

Commit 6f6a8fd

Browse files
aeiwzaeiwz
committed
Add benchmarks result
1 parent 4774fb4 commit 6f6a8fd

5 files changed

Lines changed: 1241 additions & 0 deletions

File tree

‎manuscript/benchmark/Result/PERFORMANCE.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# metbit Performance Benchmark Report
2+
3+
**Generated:** 2026-06-20T03:22:23Z
4+
**Python:** 3.14.5
5+
**Platform:** Darwin 25.5.0 (arm64)
6+
**CPU cores:** 15
7+
8+
## Backend Status
9+
10+
| Backend | Available |
11+
|---------|-----------|
12+
| Native C extension | yes |
13+
| OpenMP parallelism | no |
14+
| GPU (CuPy/PyTorch) | no |
15+
16+
_Speedup is computed as: `baseline_min_ms / new_min_ms` (higher = faster)._
17+
_Memory: `baseline_peak_mb / new_peak_mb` via tracemalloc (Python-visible allocations)._
18+
_Trend: vs previous benchmark run (previous data available)._
19+
20+
## VIP Score Computation
21+
22+
The Python loop over features was the original implementation. NumPy vectorized replaces it with a single BLAS matrix multiply. C dispatch adds OpenMP parallelism over features.
23+
24+
| Dataset | Implementation | Min (ms) | Mean (ms) | Max (ms) | Speedup vs baseline | Trend |
25+
|---------|----------------|----------|-----------|----------|---------------------|-------|
26+
| 80 x 1,000 | Python loop | 6.5 | 6.6 | 6.8 | 1.0x (baseline) | new |
27+
| 80 x 1,000 | NumPy vectorized | 0.0 | 0.0 | 0.0 | 309.5x | new |
28+
| 80 x 1,000 | C dispatch | 0.0 | 0.0 | 0.0 | 1352.6x | new |
29+
| 80 x 5,000 | Python loop | 54.3 | 54.4 | 54.8 | 1.0x (baseline) | new |
30+
| 80 x 5,000 | NumPy vectorized | 0.1 | 0.1 | 0.1 | 801.3x | new |
31+
| 80 x 5,000 | C dispatch | 0.0 | 0.0 | 0.0 | 2701.5x | new |
32+
| 80 x 20,000 | Python loop | N/A | N/A | N/A | - | - |
33+
| 80 x 20,000 | NumPy vectorized | 0.2 | 0.3 | 0.3 | N/A | new |
34+
| 80 x 20,000 | C dispatch | 0.1 | 0.1 | 0.1 | N/A | new |
35+
36+
## Pearson Correlation (STOCSY kernel)
37+
38+
The baseline materialises a full centred matrix copy (O(n*p) memory). The new implementations avoid this copy entirely.
39+
40+
| Dataset | Implementation | Min (ms) | Mean (ms) | Peak RAM (MB) | Speedup vs baseline | Memory vs baseline |
41+
|---------|----------------|----------|-----------|---------------|---------------------|---------------------|
42+
| 200 x 10,000 | Full-copy NumPy (baseline) | 1.1 | 1.1 | 16.3 | 1.0x (baseline) | baseline |
43+
| 200 x 10,000 | C dispatch f64 | 0.9 | 0.9 | 0.3 | 1.2x | 51x less |
44+
| 200 x 10,000 | C dispatch f32 | 0.7 | 0.8 | 0.3 | 1.5x | 51x less |
45+
| 200 x 10,000 | Chunked NumPy (no C) | 1.1 | 1.2 | 16.5 | 1.0x | 1x less |
46+
| 500 x 30,000 | Full-copy NumPy (baseline) | 8.3 | 8.7 | 121.0 | 1.0x (baseline) | baseline |
47+
| 500 x 30,000 | C dispatch f64 | 6.7 | 6.7 | 1.0 | 1.2x | 126x less |
48+
| 500 x 30,000 | C dispatch f32 | 6.0 | 6.0 | 1.0 | 1.4x | 126x less |
49+
| 500 x 30,000 | Chunked NumPy (no C) | 952.1 | 976.0 | 135.5 | 0.0x | 1x less |
50+
51+
## Column Variance (feature pre-selection)
52+
53+
Per-column sample variance used by `feature_preselection()`. C backend avoids creating the centred copy.
54+
55+
| Dataset | Implementation | Min (ms) | Mean (ms) | Peak RAM (MB) | Speedup vs baseline | Memory vs baseline |
56+
|---------|----------------|----------|-----------|---------------|---------------------|---------------------|
57+
| 200 x 50,000 | NumPy (baseline) | 5.8 | 6.1 | 80.8 | 1.0x (baseline) | baseline |
58+
| 200 x 50,000 | C dispatch f64 | 3.5 | 3.5 | 0.8 | 1.7x | 101x less |
59+
| 200 x 50,000 | C dispatch f32 | 3.0 | 3.1 | 0.8 | 1.9x | 101x less |
60+
| 500 x 100,000 | NumPy (baseline) | 30.3 | 31.8 | 401.6 | 1.0x (baseline) | baseline |
61+
| 500 x 100,000 | C dispatch f64 | 17.6 | 17.7 | 1.6 | 1.7x | 251x less |
62+
| 500 x 100,000 | C dispatch f32 | 15.5 | 15.9 | 1.6 | 2.0x | 251x less |
63+
64+
## Feature Pre-selection
65+
66+
Full `feature_preselection()` call including percentile threshold and mask, compared to a raw NumPy variance computation.
67+
68+
| Dataset | Implementation | Min (ms) | Mean (ms) | Max (ms) | Speedup vs baseline | Trend |
69+
|---------|----------------|----------|-----------|----------|---------------------|-------|
70+
| 100 x 50,000 | Raw NumPy var (baseline) | 3.5 | 3.6 | 3.7 | 1.0x (baseline) | new |
71+
| 100 x 50,000 | feature_preselection dispatch | 7.5 | 7.8 | 8.2 | 0.5x | new |
72+
| 200 x 100,000 | Raw NumPy var (baseline) | 11.3 | 11.5 | 11.6 | 1.0x (baseline) | new |
73+
| 200 x 100,000 | feature_preselection dispatch | 30.2 | 30.8 | 32.0 | 0.4x | new |
74+
75+
## STOCSY: ChunkedSTOCSY vs Standard
76+
77+
`ChunkedSTOCSY` bounds memory to O(n * chunk_size). This table shows the overhead vs the standard single-pass kernel.
78+
79+
| Dataset | Implementation | Min (ms) | Mean (ms) | Max (ms) | Speedup vs baseline | Trend |
80+
|---------|----------------|----------|-----------|----------|---------------------|-------|
81+
| 50 x 2,000 | Standard STOCSY | 6.1 | 6.7 | 7.6 | 1.0x (baseline) | new |
82+
| 50 x 2,000 | ChunkedSTOCSY | 0.2 | 0.2 | 0.2 | 26.6x | new |
83+
| 100 x 5,000 | Standard STOCSY | 26.8 | 28.2 | 29.2 | 1.0x (baseline) | new |
84+
| 100 x 5,000 | ChunkedSTOCSY | 0.9 | 0.9 | 0.9 | 31.1x | new |
85+
86+
## OPLS-DA Full Pipeline (fit + VIP)
87+
88+
Full `opls_da.fit()` + `vip_scores()` workflow. float32 path uses half the peak memory with negligible Q2 difference.
89+
90+
| Dataset | Implementation | Min (ms) | Mean (ms) | Max (ms) | Speedup vs baseline | Trend |
91+
|---------|----------------|----------|-----------|----------|---------------------|-------|
92+
| 60 x 300 | float64 pipeline | 27.5 | 29.3 | 35.8 | 1.0x (baseline) | new |
93+
| 60 x 300 | float32 pipeline | 37.6 | 37.9 | 38.3 | 0.7x | new |
94+
| 100 x 500 | float64 pipeline | 377.4 | 383.0 | 400.8 | 1.0x (baseline) | new |
95+
| 100 x 500 | float32 pipeline | 196.9 | 198.8 | 205.0 | 1.9x | new |
96+
97+
## Notes
98+
99+
- All times are wall-clock (min over 5 repetitions). CPU frequency scaling
100+
and cache effects cause run-to-run variability of ±10-20%.
101+
- 'N/A' in VIP loop rows indicates the benchmark was skipped (p too large for
102+
the loop to complete in a reasonable time).
103+
- Speedup ratios > 1.0x mean the new implementation is faster.
104+
- The C extension and OpenMP parallel path are auto-selected by `_native.py`
105+
based on dataset size. Thresholds: n*p > 10M -> parallel path.

‎manuscript/benchmark/Result/benchmark-report.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# metbit Full-Scale Benchmark Report
2+
3+
Generated: 2026-06-20T03:22:23Z
4+
Python: 3.14.5
5+
Platform: Darwin 25.5.0 (arm64)
6+
CPU cores: 15
7+
8+
## Backend
9+
10+
- Native C extension: yes
11+
- OpenMP threads: 0
12+
- GPU CuPy: no
13+
- GPU Torch: no
14+
15+
## Highlights
16+
17+
- `vip_scores` `C dispatch` at 80x5,000: 2701.5x speedup (0.020 ms vs 54.256 ms baseline).
18+
- `vip_scores` `C dispatch` at 80x1,000: 1352.6x speedup (0.005 ms vs 6.538 ms baseline).
19+
- `vip_scores` `NumPy vectorized` at 80x5,000: 801.3x speedup (0.068 ms vs 54.256 ms baseline).
20+
- `vip_scores` `NumPy vectorized` at 80x1,000: 309.5x speedup (0.021 ms vs 6.538 ms baseline).
21+
- `stocsy` `ChunkedSTOCSY` at 100x5,000: 31.1x speedup (0.862 ms vs 26.795 ms baseline).
22+
- `stocsy` `ChunkedSTOCSY` at 50x2,000: 26.6x speedup (0.230 ms vs 6.134 ms baseline).
23+
- `column_variances` `C dispatch f32` at 500x100,000: 2.0x speedup (15.470 ms vs 30.326 ms baseline, peak 1.6 MB).
24+
- `column_variances` `C dispatch f32` at 200x50,000: 1.9x speedup (3.024 ms vs 5.846 ms baseline, peak 0.8 MB).
25+
- `opls_da_pipeline` `float32 pipeline` at 100x500: 1.9x speedup (196.924 ms vs 377.403 ms baseline).
26+
- `column_variances` `C dispatch f64` at 500x100,000: 1.7x speedup (17.597 ms vs 30.326 ms baseline, peak 1.6 MB).
27+
28+
## Result Table
29+
30+
| Benchmark | Dataset | Implementation | Min ms | Mean ms | Peak MB | Speedup |
31+
|---|---:|---|---:|---:|---:|---:|
32+
| vip_scores | 80x1,000 | Python loop | 6.538 | 6.637 | 0.000 | baseline |
33+
| vip_scores | 80x1,000 | NumPy vectorized | 0.021 | 0.026 | 0.000 | 309.5x |
34+
| vip_scores | 80x1,000 | C dispatch | 0.005 | 0.006 | 0.000 | 1352.6x |
35+
| vip_scores | 80x5,000 | Python loop | 54.256 | 54.422 | 0.000 | baseline |
36+
| vip_scores | 80x5,000 | NumPy vectorized | 0.068 | 0.072 | 0.000 | 801.3x |
37+
| vip_scores | 80x5,000 | C dispatch | 0.020 | 0.020 | 0.000 | 2701.5x |
38+
| vip_scores | 80x20,000 | Python loop | N/A | N/A | 0.000 | baseline |
39+
| vip_scores | 80x20,000 | NumPy vectorized | 0.246 | 0.257 | 0.000 | N/A |
40+
| vip_scores | 80x20,000 | C dispatch | 0.077 | 0.081 | 0.000 | N/A |
41+
| pearson_columns | 200x10,000 | Full-copy NumPy (baseline) | 1.091 | 1.147 | 16.323 | baseline |
42+
| pearson_columns | 200x10,000 | C dispatch f64 | 0.901 | 0.924 | 0.320 | 1.2x |
43+
| pearson_columns | 200x10,000 | C dispatch f32 | 0.729 | 0.756 | 0.320 | 1.5x |
44+
| pearson_columns | 200x10,000 | Chunked NumPy (no C) | 1.140 | 1.157 | 16.485 | 1.0x |
45+
| pearson_columns | 500x30,000 | Full-copy NumPy (baseline) | 8.335 | 8.659 | 120.965 | baseline |
46+
| pearson_columns | 500x30,000 | C dispatch f64 | 6.711 | 6.739 | 0.960 | 1.2x |
47+
| pearson_columns | 500x30,000 | C dispatch f32 | 5.964 | 6.024 | 0.960 | 1.4x |
48+
| pearson_columns | 500x30,000 | Chunked NumPy (no C) | 952.100 | 975.994 | 135.533 | 0.0x |
49+
| column_variances | 200x50,000 | NumPy (baseline) | 5.846 | 6.110 | 80.801 | baseline |
50+
| column_variances | 200x50,000 | C dispatch f64 | 3.494 | 3.535 | 0.800 | 1.7x |
51+
| column_variances | 200x50,000 | C dispatch f32 | 3.024 | 3.100 | 0.800 | 1.9x |
52+
| column_variances | 500x100,000 | NumPy (baseline) | 30.326 | 31.775 | 401.601 | baseline |
53+
| column_variances | 500x100,000 | C dispatch f64 | 17.597 | 17.717 | 1.600 | 1.7x |
54+
| column_variances | 500x100,000 | C dispatch f32 | 15.470 | 15.854 | 1.600 | 2.0x |
55+
| feature_preselection | 100x50,000 | Raw NumPy var (baseline) | 3.452 | 3.571 | 0.000 | baseline |
56+
| feature_preselection | 100x50,000 | feature_preselection dispatch | 7.515 | 7.799 | 0.000 | 0.5x |
57+
| feature_preselection | 200x100,000 | Raw NumPy var (baseline) | 11.346 | 11.477 | 0.000 | baseline |
58+
| feature_preselection | 200x100,000 | feature_preselection dispatch | 30.164 | 30.794 | 0.000 | 0.4x |
59+
| stocsy | 50x2,000 | Standard STOCSY | 6.134 | 6.688 | 0.000 | baseline |
60+
| stocsy | 50x2,000 | ChunkedSTOCSY | 0.230 | 0.239 | 0.000 | 26.6x |
61+
| stocsy | 100x5,000 | Standard STOCSY | 26.795 | 28.217 | 0.000 | baseline |
62+
| stocsy | 100x5,000 | ChunkedSTOCSY | 0.862 | 0.885 | 0.000 | 31.1x |
63+
| opls_da_pipeline | 60x300 | float64 pipeline | 27.506 | 29.302 | 0.000 | baseline |
64+
| opls_da_pipeline | 60x300 | float32 pipeline | 37.597 | 37.878 | 0.000 | 0.7x |
65+
| opls_da_pipeline | 100x500 | float64 pipeline | 377.403 | 382.985 | 0.000 | baseline |
66+
| opls_da_pipeline | 100x500 | float32 pipeline | 196.924 | 198.850 | 0.000 | 1.9x |
67+
68+
## Notes
69+
70+
- `N/A` means the baseline was skipped, usually because the large Python-loop baseline would take too long.
71+
72+
## Artifacts
73+
74+
- Detailed report: `reports/PERFORMANCE.md`
75+
- Raw data: `reports/benchmark_results.json`
76+
- Previous run snapshot: `reports/benchmark_results_prev.json`

0 commit comments

Comments
 (0)