ignore manuscript

aeiwz · aeiwz · commit fdd415f909d2 · 2026-07-12T01:17:41.000+07:00
diff --git a/.gitignore b/.gitignore
@@ -54,3 +54,5 @@ ENV/
 /manuscript/benchmark/data/
 /manuscript/benchmark/results/
 manuscript/incob2026/~$cob2026_manuscript.docx
+/manuscript
+/manuscript
diff --git a/manuscript/incob2026_manuscript.md b/manuscript/incob2026_manuscript.md
@@ -43,6 +43,8 @@ The present manuscript describes the design and implementation of metbit version
 
 metbit (version 9.0.0) is implemented in Python 3.10 and later. It is organized as a flat-namespace package with distinct modules corresponding to stages of the NMR metabolomics workflow: `nmr_preprocessing`, `normalization`, `alignment`, `pca`, `opls_da`, `stocsy`, and `utils`, together with a new `large_scale` module and an auto-dispatch compute backend. Module-level classes accept pandas DataFrames or NumPy arrays as primary inputs, enabling direct compatibility with standard Python data-science workflows. Each modeling class returns fitted objects that expose plots, statistics, and intermediate arrays, so downstream analyses can be driven by the same in-memory objects that generated visualizations. The package is released under the MIT license, versioned through Git and published on PyPI, and its documentation is built and hosted at https://metbit-docs.vercel.app.
 
+metbit was first released on PyPI as version 1.0.0 in February 2024, providing initial NMR preprocessing, normalization, PCA, and OPLS-DA functionality. Subsequent releases through the 1.x-8.x series (227 tagged versions as of this writing) incrementally added icoshift alignment, STOCSY, VIP scoring, the interactive Plotly and Dash interfaces, and expanded test coverage. Version 9.0.0 (19 June 2026), the subject of this manuscript, is a major release that introduces the four-tier auto-dispatch compute backend, the `large_scale` module, and the current 305-test suite described in Section 2.10, targeting the memory and throughput constraints of large-cohort datasets. The complete release history is documented in the project changelog at https://github.com/aeiwz/metbit/blob/main/CHANGELOG.md.
+
 ### 2.2 NMR Preprocessing
 
 Raw Bruker FID directories are handled by the `nmr_preprocessing` module. The pipeline removes digital filter artifacts, applies zero filling to the next power of two, executes the Fourier transform, performs automated phase correction using entropy minimization (Chen et al., 2002), applies asymmetrically reweighted penalized least squares (arPLS) baseline correction (Baek et al., 2015) as implemented in pybaselines (Erb, 2022), and calibrates the chemical-shift axis to an internal reference. Users who have already processed their spectra in instrument software can begin directly at the normalization or modeling stage by providing a sample-by-variable pandas DataFrame. This flexibility allows metbit to be incorporated into existing laboratory workflows without repeating upstream steps.
@@ -65,7 +67,11 @@ STOCSY (Cloarec et al., 2005) correlates the intensity of every spectral variabl
 
 ### 2.7 Interactive Visualization and Dash Applications
 
-All visualizations are produced with Plotly, delivering HTML-embedded interactive outputs that can be shared without a running Python server. Four local Dash applications are included: a STOCSY explorer, a spectral annotation interface for chemical-shift regions of interest, a peak-selection tool, and a model-comparison dashboard. These applications are launched with a single function call and run on localhost, making them accessible in environments where server deployment is not feasible.
+All visualizations are produced with Plotly, delivering HTML-embedded interactive outputs that can be shared without a running Python server. Every scores, loadings, and STOCSY figure supports pan, zoom, legend-based group toggling, and on-hover tooltips reporting the sample identifier, group label, and coordinate values for the point under the cursor (Figure 1). Four local Dash applications are included: a STOCSY explorer, a spectral annotation interface for chemical-shift regions of interest, a peak-selection tool, and a model-comparison dashboard. These applications are launched with a single function call and run on localhost, making them accessible in environments where server deployment is not feasible.
+
+![metbit interactive-visualization workflow on the MTBLS1 case study (Section 2.13): (A) preprocessed spectral overlay, (B) PCA scores, (C) OPLS-DA scores, (D) VIP scores, and (E) STOCSY anchored at the top-VIP resonance. Callouts reproduce the content of the live on-hover tooltip for a representative sample or point; in the interactive HTML/Dash output this information appears automatically as the cursor moves over any marker, trace, or point, and groups can be toggled on or off from the legend.](figures/figure_interactive_workflow.png)
+
+**Figure 1.** Snapshot sequence of metbit's interactive Plotly outputs across the analytical workflow. Each panel is generated by the corresponding metbit plotting method (`SpectraPlot.overlay`, `pca.plot_pca_scores`, `opls_da.plot_oplsda_scores`, `opls_da.vip_plot`, `STOCSY`) on the MTBLS1 case-study data. The underlying figure objects are fully interactive (pan, zoom, hover, legend toggling) when rendered in a Jupyter notebook, exported HTML file, or Dash application; this static export illustrates the hover-tooltip content and the sequence in which the outputs are typically produced.
 
 ### 2.8 Large-Scale Compute Backend
 
@@ -91,13 +97,13 @@ Reproducible timing measurements for version 9.0.0 were collected using `scripts
 
 ### 2.12 Synthetic Workflow Example
 
-To demonstrate the matrix-based workflow in version 9.0.0, we used a two-group synthetic NMR dataset distributed with metbit. The dataset contains 120 samples, four time points, and 65,536 variables spanning 0.0–9.0 ppm. Group differences were introduced into selected spectral regions to produce a controllable two-class structure. Workflow timing for this specific dataset was not independently re-measured for this manuscript; performance characteristics at representative dimensions are instead reported from the reproducible benchmarks described in Section 2.11. All benchmarks were executed on an Apple M5 Pro workstation (arm64 architecture, 15 logical CPU cores, 24 GB unified RAM) running macOS Tahoe 26.5.1 (Darwin kernel 25.5.0) under Python 3.14.5, NumPy 2.4.6, SciPy 1.17.1, scikit-learn 1.8.0, and pandas 2.3.3, with the metbit native C extension active (single-threaded; OpenMP and GPU acceleration not available on this configuration). The benchmark dimensions (80–500 samples, 1,000–100,000 features) bracket the synthetic example's sample count and span a range of feature counts relevant to typical NMR and LC-MS metabolomics datasets.
+To demonstrate the matrix-based workflow in version 9.0.0, we used a two-group synthetic NMR dataset distributed with metbit. The dataset contains 120 samples, four time points, and 65,536 variables spanning 0.0–9.0 ppm. Group differences were introduced into selected spectral regions to produce a controllable two-class structure. Workflow timing for this specific dataset was not independently re-measured for this manuscript; performance characteristics at representative dimensions are instead reported from the reproducible benchmarks described in Section 2.11, which were run on the same environment. The benchmark dimensions (80–500 samples, 1,000–100,000 features) bracket the synthetic example's sample count and span a range of feature counts relevant to typical NMR and LC-MS metabolomics datasets.
 
 ### 2.13 MTBLS1 Technical Case Study
 
 To demonstrate operation on public raw data, metbit was applied to MTBLS1, a human urine <sup>1</sup>H NMR study available through MetaboLights (Haug et al., 2020; Salek et al., 2007). The analysis included 132 successfully processed spectra: 48 labeled as diabetes mellitus and 84 as controls. The study metadata describe repeated collections from 30 patients and 12 healthy participants. Disease status was also completely associated with acquisition series: diabetes spectra used the `ADG10003u` prefix and were acquired in June 2004, whereas control spectra used `ADG19007u` and were acquired in May 2004. Disease and acquisition batch could therefore not be separated.
 
-Raw Bruker FID archives were processed with digital-filter removal, zero filling, Fourier transformation, ACME phase correction (Chen et al., 2002), arPLS baseline correction (Baek et al., 2015), TSP calibration, and icoshift alignment. PQN normalization was then applied to the complete spectral matrix, followed by Pareto-scaled PCA and a two-component OPLS-DA model. The exploratory implementation used sample-level seven-fold cross-validation; it did not group repeated observations by participant, and normalization and alignment were not refitted within each training fold. These choices can leak information across folds and produce optimistic statistics. Accordingly, R<sup>2</sup>Y, Q<sup>2</sup>, predictive-score AUROC, and VIP rankings are reported only to document software output. They are not interpreted as estimates of disease discrimination or generalizability. The permutation output was excluded because the implemented test did not evaluate the same OPLS Q<sup>2</sup> statistic. Analyses were performed on an Apple M5 Pro workstation (arm64, 15 logical CPU cores, 24 GB unified RAM) running macOS Tahoe 26.5.1 (Darwin 25.5.0) under Python 3.14.5, NumPy 2.4.6, SciPy 1.17.1, scikit-learn 1.8.0, pandas 2.3.3, and metbit 9.0.0 with the native C extension active.
+Raw Bruker FID archives were processed with digital-filter removal, zero filling, Fourier transformation, ACME phase correction (Chen et al., 2002), arPLS baseline correction (Baek et al., 2015), TSP calibration, and icoshift alignment. PQN normalization was then applied to the complete spectral matrix, followed by Pareto-scaled PCA and a two-component OPLS-DA model. The exploratory implementation used sample-level seven-fold cross-validation; it did not group repeated observations by participant, and normalization and alignment were not refitted within each training fold. These choices can leak information across folds and produce optimistic statistics. Accordingly, R<sup>2</sup>Y, Q<sup>2</sup>, predictive-score AUROC, and VIP rankings are reported only to document software output. They are not interpreted as estimates of disease discrimination or generalizability. The permutation output was excluded because the implemented test did not evaluate the same OPLS Q<sup>2</sup> statistic. Analyses were performed on the same hardware and software environment described in Section 2.11, using metbit 9.0.0 with the native C extension active.
 
 ---
 
@@ -195,7 +201,7 @@ The version 9.0.0 performance benchmarks demonstrate two complementary improveme
 
 The ChunkedSTOCSY class addresses a related constraint. At large feature counts, the standard STOCSY implementation constructs a full Plotly figure object with one data point per feature, which itself becomes a bottleneck in Python object allocation. ChunkedSTOCSY separates correlation computation from visualization and processes features in configurable batches, producing bit-identical results to the standard kernel while bounding peak memory and providing a 47-fold throughput advantage at 100 samples and 5,000 features. Users working with datasets at scales where the standard STOCSY completes in under a second will notice no meaningful difference; the ChunkedSTOCSY path becomes important when feature counts exceed approximately 100,000 on a typical workstation.
 
-The reproducible benchmark methodology introduced in version 9.0.0 represents a deliberate improvement over the single-run timing reported in the original manuscript, which did not record repeated-run dispersion, warm-up conditions, or the complete software environment. The current benchmarks use five repetitions with one discarded warm-up, report minimum times alongside peak memory, archive results to a JSON file for trend comparison, and commit the benchmark script and environment specification to the repository. This approach is consistent with the criteria for performance reproducibility identified in the Discussion of this manuscript and in recent guidance on scientific computing benchmarking. Users wishing to reproduce or extend the benchmarks can run `python scripts/perf_report.py` from the repository root after installation. The reported values should nonetheless be interpreted as characterizing one specific hardware and software environment; performance on other systems will differ, particularly for the OpenMP and GPU paths that were not exercised in the current benchmarks.
+The reproducible benchmark methodology introduced in version 9.0.0 represents a deliberate improvement over the ad hoc single-run timing used during earlier metbit development, which did not record repeated-run dispersion, warm-up conditions, or the complete software environment. The current benchmarks use five repetitions with one discarded warm-up, report minimum times alongside peak memory, archive results to a JSON file for trend comparison, and commit the benchmark script and environment specification to the repository. This approach is consistent with the criteria for performance reproducibility identified in the Discussion of this manuscript and in recent guidance on scientific computing benchmarking. Users wishing to reproduce or extend the benchmarks can run `python scripts/perf_report.py` from the repository root after installation. The reported values should nonetheless be interpreted as characterizing one specific hardware and software environment; performance on other systems will differ, particularly for the OpenMP and GPU paths that were not exercised in the current benchmarks.
 
 The test suite of 305 tests addresses a gap in earlier releases, which lacked systematic verification of statistical validity, dispatch routing, and numerical equivalence across backends. The AB/AA statistical validity paradigm—verifying that the model discriminates genuine signal and fails to discriminate noise—provides a data-driven correctness criterion complementary to unit tests of individual functions. The memory-efficiency assertions using `tracemalloc` verify the primary design claim of the C kernels: that peak allocation scales with O(p) working arrays rather than O(n × p) input data. These tests are designed to catch regressions that would silently restore the memory-expensive code path without breaking numerical outputs.
 
