Migrate/authored - #9
Merged
Merged
Conversation
…he reuse Machine-learning correctness: - QProfiler fit a *fresh* MinMaxScaler on X_test, normalizing the test set with test-set statistics. This both used held-out information and trained/evaluated the model under different transforms. Add qbiocode.scale_train_test (one scaler fit on train, applied to test) and use it. QProfiler metrics from before this commit are not comparable to metrics after it. - train_test_split had no random_state, so `seed` was silently ignored for splitting and iterations were irreproducible -- despite an inline comment claiming the splits were seed-based. Use random_state = seed + iter. Embedding crashes: - get_embeddings(n_components=None), the documented default, raised TypeError: '<=' not supported between 'NoneType' and 'int'. Default to X_train.shape[1] as documented. - `spectral` raised AttributeError: SpectralEmbedding has no 'transform' on every run. Fit transductively over combined train+test rows and slice back; only feature structure participates, so no label leakage is introduced. Quantum sessions and caching: - Close Qiskit Runtime sessions in a finally block in embed.pqk and learning.compute_pqk; an exception previously leaked the session. - PQK projection cache filenames omitted encoding/entanglement/reps/primitive, so re-running with a different feature map into the same pqk_projection_dir silently reloaded and reported the previous run's projections. Key the cache on a digest of those parameters, and validate cached width (3 x feat_dimension) in addition to row count. Also thread n_neighbors from config through to get_embeddings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tier the dependency lists and make pyproject.toml the only place project
metadata is declared. Adopts internal's requirements/ layout, corrected.
Dependency tiers (requirements/):
requirements-base.txt core runtime -- the single source of truth, read by
pyproject.toml via [tool.setuptools.dynamic]
requirements-quvine.txt the new [quvine] extra
requirements-docs.txt the [docs] extra
requirements.txt everything, for development environments
The root requirements.txt becomes a pointer to requirements/requirements.txt so
the `pip install -r requirements.txt` documented in CONTRIBUTING.md,
docs/source/installation.md, ISMB_2026.rst and the PR template keeps working.
Fixes carried in this commit:
* Three dependencies were imported but never declared. pyyaml is imported by
utils/generate_qml_configs.py and apps/qprofiler/qprofiler_batchmode.py (which
backs the qprofiler-batch console script); matplotlib by five modules; joblib
by evaluation/model_run.py and qprofiler_batchmode.py. Installs only worked
because seaborn and optuna pulled them in transitively.
* MANIFEST.in referenced apps/qprofiler/configs, which does not exist -- the
configs live under qbiocode/apps/qprofiler/configs -- so the sdist shipped no
Hydra config for QProfiler. packages.find likewise carried an apps* include
glob matching no package.
* pandoc was declared as a pip dependency of [docs]. Pandoc is a system binary;
the PyPI distribution of that name does not provide it. The platform install
commands are documented in requirements-docs.txt instead.
* setup.py's 189 lines of duplicated metadata are replaced by internal's 33-line
shim that passes no arguments. The old version re-declared name, version,
dependencies, extras, classifiers and entry points, and split runtime from
docs dependencies by scanning requirements.txt for a "# Documentation" comment.
* [all] is now "qbiocode[apps,quvine,docs,dev]" rather than a hand-maintained
copy of every other extra, which nothing kept honest.
* tensorflow is dropped. Nothing in qbiocode/, the tutorials or the docs imports
TensorFlow or Keras; the only reference left was a stale autodoc_mock_imports
entry. compute_autoencoder.py, the one module that could have needed it, is
written against PyTorch. Removes roughly 600 MB from a default install.
* .gitignore matched data/ and results/ unanchored, excluding a directory of
either name at any depth -- including docs/source/tutorials/QProfiler/data/,
which holds the committed .h5ad and sc_binary/*.csv fixtures the published
notebooks read. Since git cannot re-include a file under an excluded
directory, those fixtures were addable only because they were already tracked.
Both patterns are now anchored to the repository root.
* Stopped tracking generated QProfiler output under docs/source/tutorials/
QProfiler/ (ModelResults.csv, RawDataEvaluation.csv, results.pkl, and 500 KB
of pqk_projections/*.npy). The notebooks write these and read them back in the
same run, and nbsphinx_execute = 'never' means the docs build never executes
them. The cached projections were additionally unreachable after Phase 1's
PQK cache-key fix, their filenames predating the feature-map fingerprint.
The [quvine] extra is all-or-nothing, as decided: one extra carrying the whole
QuVINE dependency set. setuptools<81 is pinned inside it because node2vec
imports the pkg_resources module that setuptools>=81 removed, and gensim>=4.4
because older gensim needs scipy.triu (removed in scipy 1.13) and would force
numpy<2.0, conflicting with qiskit-machine-learning==0.9.0.
The quvine console script and its package-data entry are deliberately NOT added
here -- qbiocode.apps.quvine does not exist until Phase 3, and installing an
entry point for a missing module would leave a broken `quvine` command.
tests/test_requirements_consistency.py (10 tests) enforces the invariants: the
quvine/docs extras match their requirements files, dynamic dependencies point at
the shipped base file, build tooling stays out of the runtime list, [all] stays a
union, no pandoc, no tensorflow, and setup.py declares no metadata.
Verified: 23 tests pass; python -m build produces 31 unconditional Requires-Dist
entries with no setuptools and no tensorflow; the sdist ships
requirements/requirements-base.txt; the wheel ships the qprofiler configs and
exactly three entry points; pip install --dry-run '.[all]' resolves every tier
(exit 0); qprofiler/qsage/qprofiler-batch --help all exit 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings QuVINE (Quantum View-based Network Embeddings) and the graph-complexity metrics over from the internal repository: 61 modules under qbiocode/apps/quvine/, qbiocode/evaluation/graph_evaluation.py exposed as qbiocode.evaluate_graph, and docs/source/apps/quvine.rst wired into the apps toctree. The decision to ship one all-or-nothing [quvine] extra rather than a plain dependency shapes everything else here. `import qbiocode` and every classical embedding must keep working with the extra absent, so: * qbiocode/apps/quvine/_deps.py resolves each optional dependency through require_module() and raises QuvineDependencyError -- naming the method, the extra, the pip command and the missing distribution -- instead of a bare ModuleNotFoundError traceback. It subclasses ImportError so the Phase 4 routing probe (`except ImportError`) keeps working. Wired into walks/ctqw.py, walks/dtqw.py, baselines/node2vec.py and embedding/word2vec.py. * api/__init__.py is lazy. It eagerly imported core/config/sgns/targets, so reaching the stdlib-only resolve_method dragged in omegaconf and the rest of the extra; that only appeared to work because hydra-core pulls omegaconf in transitively. The CLI likewise imports embed at its point of use, so --help and --list-methods run on a bare install. Four defects fixed in the port rather than carried over: * The five fused names (quvine_fused, quvine, fused, ...) were handled inside embed() but absent from the alias tables, so resolve_method raised KeyError and the CLI rejected its own default --method. Fused is now a first-class kind in api/aliases.py, list_methods() reports 83 names, and the CLI validates through resolve_method so its message and the dispatchable set cannot drift. * walks/, corpus/, utils/ and configs/ had no __init__.py, so packages.find skipped them: the modules imported fine from a source checkout but were missing from a built wheel. Verified by importing them out of an unpacked wheel. * torch was declared in both the base set and the extra. qbiocode.embeddings imports it eagerly (ConvAutoencoder), so a bare install already needs it -- a missing torch is a broken install, not a missing extra, and the [quvine] hint would be wrong advice. Now base-only. * word2vec.py advised `pip install gensim==4.3.0 scipy==1.11.0` on failure, a downgrade that forces numpy<2.0 and conflicts with qiskit-machine-learning==0.9.0. Replaced with the extra; the unused GENSIM_AVAILABLE flag went with it. Two false claims corrected: quvine/__init__.py and docs/source/apps/quvine.rst both said QuVINE was "available on a plain install with no extra install step". The docs Tutorials section links the tutorials index rather than the QuVINE notebook, which arrives in Phase 7 -- a :doc: role pointing at a file that does not exist yet would fail the `sphinx-build -W` run added in Phase 8. pyproject.toml gains the `quvine` console script and the quvine configs package-data, both deliberately deferred from Phase 2 until the module existed. tests/test_quvine_packaging.py (18 tests) locks the invariants down: a subprocess with every [quvine] module blocked still imports qbiocode and resolves method names; the _deps table matches requirements-quvine.txt; an AST scan (including require_module string arguments, which an import scan misses) finds no undeclared third-party import under qbiocode/; every quvine directory is a package; every listed method resolves and every resolvable kind has a dispatch branch; and the CLI's default method is one the CLI accepts. 41 tests pass. Wheel ships 66 quvine files, the packaged config.yaml, graph_evaluation.py and 4 console scripts; sdist metadata has 31 unconditional Requires-Dist with torch present and no setuptools leak. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_embeddings("quvine_rwr", X_train, X_test, n_components=8) now behaves
exactly like get_embeddings("pca", ...) -- same call shape, same (Z_train,
Z_test) return, any of the 83 QuVINE method names accepted, reachable from
QProfiler's existing embeddings: config list.
Public surface:
- SKLEARN_METHODS / QUVINE_HEADLINE_METHODS / QUVINE_METHODS name what is
available; QUVINE_METHODS is empty rather than raising when the [quvine]
extra is absent.
- is_transductive(name) reports whether a method sees test features at embed
time, and get_embeddings emits one UserWarning per call for those methods
saying that test features shape the geometry while test labels never do.
- Boundary validation: unknown names, non-string embedding, and non-integer /
zero / negative / too-wide n_components all raise ValueError naming the
parameter, the value and the accepted set, with close-name suggestions.
Three defects had to be fixed before any registry method could run at all.
qbiocode/apps/quvine/data/ was never committed to the internal repository -- an
unanchored `data/` rule in its .gitignore matched the directory at every depth,
so it appears nowhere in that history. embedding/quantum_filters.py imports it,
and through it the adapter module that builds the registry, so all 69 registry
methods died with ModuleNotFoundError. data/subgraph.py is reimplemented from
its two call sites (bounded-radius ego-net expansion is well determined), which
restores the registry; the four modules that remain absent gate only Pipeline's
on-disk loading and the synthetic benchmark generators, and now raise
QuvineDataUnavailableError naming the module and the feature. Pipeline imports
again.
Optional dependencies were resolved at import time in walks/ctqw.py,
walks/dtqw.py and baselines/node2vec.py, so failures were attributed to
whichever module was imported first: an RWR-only run blamed CTQW's hiperwalk,
and node2vec -- swallowed by baselines/__init__.py's except ImportError: pass --
left run_node2vec unbound and took netmf, appnp and graphgps down with it. All
three resolve at call time now and each names its own dependency; a test asserts
structurally that no module under apps/quvine calls require_module at module
scope. GraphGPS's hand-rolled torch-geometric message goes through the same
helper.
The registry printed to stdout (✓ netmf: 0.00 minutes) on every call and logged
a full traceback for a failure it returned to its caller to re-raise. Progress
goes through logging, the traceback is available at DEBUG, and the cause travels
on MethodResult.exception so api.core chains it.
tests/test_embedding_layer.py: 29 tests. 70 pass in total.
qbiocode/apps/quvine/data/ is absent from the internal repository and from its entire history: an unanchored `data/` rule in that repository's .gitignore matched the directory at every depth, so `git add` never picked it up. The source was recovered from the working tree the vendoring was done in (NetMed/QBC/QBioCode at merge-base 83192c4), which is byte-identical to the internal tree apart from license headers and two `except:` widenings -- so this is the same vintage as the rest of the ported app, not a newer or older copy. Six modules, all needing nothing beyond the base install (numpy, pandas, networkx, scipy): data_loader graph and GWAS-table loading prepare sparsification, LCC extraction, node subsampling sparsify edge sparsification primitives subgraph bounded-radius ego-nets, degree-matched subsampling random_graphs 13 synthetic generators random_graphs_extended 11 more, incl. SBM variants and expander-like graphs This restores three surfaces that were dead: - All 69 registry methods. embedding/quantum_filters.py imports data.subgraph at module scope and baselines/adapters.py imports quantum_filters, so the whole registry raised ModuleNotFoundError. - qbiocode.apps.quvine.Pipeline, which imports data_loader and prepare at module scope. - All 15 synthetic graph families in reproducibility.graph_generator, which imports the two random_graphs modules inside SyntheticGraphGenerator's constructor. Verified: each generates a non-empty graph and writes it. The QuvineDataUnavailableError / require_data_module stopgap added with Phase 4 is removed and pipeline.py and graph_generator.py revert to their upstream form (both are now byte-identical to their pre-Phase-4 state). The reimplemented data/subgraph.py is replaced by the original. Two deliberate departures from the recovered source: - graph_complexity.py is not carried over. Nothing imports it and qbiocode.evaluate_graph supersedes it. - expand_neighborhood filtered roots absent from the graph only for radius >= 1; at radius=0 it returned them verbatim and could hand back nodes the graph does not contain. Roots are now filtered first. data/__init__.py re-exports the 37 public names; MANIFEST.in ships the generators' README in the sdist. External's .gitignore anchors the rule as `/data/`, so this class of silent omission cannot recur here. 74 tests pass.
Three failure shapes, each of which shipped and each of which is worse than an
exception:
1. A fabricated value returned as a measurement. Link-prediction AUC reported 0.5
(the score of a random ranker) for an undefined metric; modularity and
path_length_ratio reported 0.0 (a real finding) when undefined; the ten
quantum-advantage metrics reported 0.0 for an empty graph, including the
rankable composite score; netmf returned np.random.randn(n, dim) * 0.01 when
its factorization failed, which the registry scored as a result. All now
report nan or raise, and summarize_link_prediction_results aggregates
nan-aware with per-metric n_defined_* counts so one undefined method no
longer erases the ones that succeeded.
2. A silent no-op reported as success. get_optimizer("L_BFGS_B") used `==` where
it meant `=`, so it built an optimizer, discarded it, and raised
UnboundLocalError for an option two public functions advertise;
get_feature_map returned its own argument for an unknown name;
scale_train_test returned unscaled data for a mistyped scaler; QProfiler
tested `'True' in args['scaling']`, which accepted 'MinMaxScalerTrue' and
crashed on `scaling: true`; compute_pqk accepted a `primitive` that changed
only the cache fingerprint; QSage aliased its metric list and then sorted it
in place, reordering the columns used to slice the input frame.
3. An error attributed to the wrong cause. Dead `except ImportError: X = None`
sentinels turned an actionable ImportError into "'NoneType' object is not
callable"; broad handlers reported a bug in their own body as a degenerate
graph. Handlers are narrowed to nx.NetworkXException / ArpackError /
LinAlgError / parser errors; the few that must stay broad name
type(e).__name__ and keep the traceback at DEBUG.
Validation moves to the boundary: get_embeddings coerces and shape-checks its
matrices, the new public check_embedding_name lets a caller validate a whole
embeddings list before any work begins, compute_pqk runs eleven checks before
touching the filesystem, model_run names the available models instead of raising
KeyError inside a joblib worker, and all three CLIs reject bad arguments before
creating an output directory.
tests/test_error_contracts.py adds 43 tests pinning this behaviour. Suite: 115
passed, 2 skipped. CHANGELOG records every number-changing fix under a
results-change warning.
import qbiocode wrote 27 entries into the global plt.rcParams, because
visualize_correlation.py assigned them at module scope and the package imports
it. Every unrelated figure the caller drew afterwards came out in Arial with no
top spine and a 600-dpi savefig default, with nothing in the call stack to
attribute it to. Those settings are now PUBLICATION_STYLE, applied per call
through plt.rc_context in a public wrapper so they are restored on the exception
path too, with publication_style() for callers who want the same look.
plot_results_correlation derived two of its three output paths with
re.sub(".pdf", "_heatmap.pdf", path). For any extension but .pdf the pattern did
not match, so the derived path equalled the original: passing corr.png left one
file where three were requested, the scatter overwritten by the clustered heatmap
and that by the non-clustered one, with nothing to say two had been lost. The
unescaped . was also a wildcard, so out/spdf_x.pdf became
out/_heatmap.pdf_x_heatmap.pdf. QSage.plot_results had the same defect and forced
.pdf regardless of the request. Both use os.path.splitext now.
plt.show() was unconditional at nine sites across three modules, hanging a batch
run under a GUI backend and warning under Agg; each is guarded on the backend
being able to display. analyze.py saved *after* showing, so a batch run stalled
with nothing written until someone closed a window, and wrote three fixed
filenames into the cwd. Every plt.close() is now plt.close(fig) on the figure
actually drawn, the touched functions use the explicit fig/ax API, and all three
functions return their figures so a notebook can compose without recomputing.
No matplotlib.use("Agg") guard: forcing a backend at import is the same hidden
global mutation this removes, and matplotlib already falls back on its own.
tests/test_plotting_hygiene.py adds 17 tests, including a subprocess check that
importing the package leaves rcParams untouched and an AST guard that fails if
any module regains import-time matplotlib styling. Suite: 132 passed, 2 skipped.
Ports internal's four QuVINE notebooks, commits the fixtures they read, and
replaces the copy-pasted path-derivation snippet each notebook carried with one
documented resolver.
New: qbiocode.tutorial_data_path()
Every single-cell notebook derived a repository root as
dirname(dirname(abspath(qbiocode.__file__))). That yields site-packages for a
normal install, so the derived fixture path pointed nowhere; the failure then
surfaced as anndata's "file not found", naming neither the QBC_DATA override
nor the directories that had been tried. Each copy also knew only one of the
four fixture directories in the tree.
tutorial_data_path() searches $QBC_DATA first, then every fixture directory of
a source checkout -- locating that checkout both from the installed package
and by walking up from the cwd, so it resolves for an editable install, for a
normal install used inside a clone, and for a notebook run from its own
subdirectory. A miss raises FileNotFoundError listing every directory tried
and the three ways to fix it. When QBC_DATA is set but lacks the file, the
file is returned from wherever it was found *and* a WARNING names both
directories: resolving quietly against a directory the caller did not name is
how a stale fixture gets read for a whole session.
Notebooks ported (each declares `pip install "qbiocode[quvine]"` up front):
tutorial/QuVINE/example_quvine.ipynb -- re-executed, 11/11 cells
tutorial/QuVINE/quvine_sc_t_vs_mono.ipynb
tutorial/QProfiler/sc_binary_quvine_2x2_qprofiler.ipynb (+ docs mirror)
docs/source/tutorials/QuVINE/quvine_sc_cd4_vs_cd8.ipynb
Fixtures: three pbmc5k .h5ad files under tutorial/QuVINE/datasets/ (13 MB).
Two of them are the provenance of the sc_binary/*.csv matrices QProfiler's
single-cell tutorial already shipped and cannot be regenerated without the raw
10x matrix plus scanpy/leidenalg. pbmc5k_small_cd4_vs_cd8.h5ad was NOT
duplicated -- the tracked copy under tutorial/QProfiler/data/ is
byte-identical and tutorial_data_path() finds it from either tree.
Fixed:
- Four notebooks imported distributions that do not exist: `from apps.qprofiler
import ...` (docs example_qprofiler), `from apps.sage.sage import ...` (docs
qsage), and `import qprofiler.qprofiler` (both QPL_example copies). Each was
an unconditional ModuleNotFoundError -- the same defect already fixed in
apps/qprofiler/cli.py.
- Eight notebooks pinned kernels that exist only on their author's machine
(venv, .env, venv_quvine, qbc-pkg); an unresolvable kernelspec fails
execution under nbclient/nbsphinx rather than falling back. All now declare
the standard python3 kernel.
- The Quantum Ensemble tutorial link was a 404: tutorials.md linked a page
Sphinx never built, because the notebook lived only under tutorial/ with no
toctree entry. Mirrored into docs/source/tutorials/QEnsemble/ (with the
helper_functions.py it imports) and added to the toctree. Every gallery link
now resolves to a built page and every built page is in the toctree.
- sc_binary_qprofiler.ipynb carried a stale warning that the PQK cache ignores
pqk_args. True before this release, no longer -- the key now includes the
feature-map fingerprint. Corrected rather than deleted, since the purge is
still worth keeping for a self-contained run.
- Both copies of sc_binary_qprofiler.ipynb now state at the top that their
outputs predate the train/test contamination fix and are not comparable to a
fresh run of the same config.
Docs: tutorials.md gains a QuVINE section as §4 (with the QProfiler-2x2 notebook
as a subsection), renumbering Ensemble/QPL/PQK-OV to 5-7, and keeps external's
QProfiler-on-single-cell-with-PQK subsection that internal had deleted.
apps/quvine.rst now links both new notebooks directly. .gitignore's
tutorial/**/data/ rule gains a comment explaining that git honours the index
over it, which is the only reason the four tracked fixtures under it survive.
Not done, deliberately:
- quvine_sc_t_vs_mono, quvine_sc_cd4_vs_cd8 and the two 2x2 copies could not
be re-executed here: they read .h5ad and anndata is not installable in this
offline environment. They ship with internal's outputs.
- sc_binary_qprofiler.ipynb diverged on both sides after the merge base;
internal's 22-cell version adds a paired Cohen's d_z forest plot and a
cross-task complexity section. Left unmerged -- merging needs re-execution,
and stitching outputs from two different runs would misrepresent them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… extras Sphinx configuration -------------------- conf.py built every path from ".": the sys.path entries, better_apidoc's template directory and its output directory. Those are correct only when sphinx-build is invoked from docs/, because conf.py is exec'd with cwd set to wherever the build started. Running `sphinx-build docs/source docs/build/html` from the repository root -- what most editor integrations do -- silently produced a build with no autodoc templates and API pages written elsewhere. All four are now derived from os.path.abspath(__file__). release was hardcoded to '0.0.1' while qbiocode/version.py declares '0.1.0', so the published docs labelled themselves a version that was never released. It is now parsed out of version.py with a regex rather than by importing the package, so it stays correct even when the docs build cannot import qbiocode. autodoc_mock_imports listed tensorflow, which appears nowhere in the tree, and omitted every [quvine] dependency, so a docs build without the extra could not import the QuVINE modules it was documenting. omegaconf mattered most: it backs QuVINE's config loading, so api/config, api/core, api/sgns, main, pipeline and utils/io were all unimportable. Base dependencies are deliberately not mocked -- mocking them would hide a broken install behind a clean docs build. run_apidoc swallowed everything with a bare `except Exception: pass`, so a missing better_apidoc, a template error or an import failure produced no output and the build carried on against whatever pages happened to be committed. A missing better_apidoc is now logged at info level (the committed pages are the intended fallback); any other failure is a Sphinx warning, so -W fails on it. API reference ------------- Nothing in any toctree pointed at api/modules, so every generated page was an orphan and the API reference was reachable only by guessing a URL. api_overview.rst now roots it with a hidden toctree entry, and gained sections for the API added in this release: Graph Embeddings (QuVINE) and a Preprocessing subsection for scale_train_test. The 23 committed pages came from two different generators -- 15 better_apidoc --separate, 8 plain sphinx-apidoc -- and there was no page for qbiocode.apps at all, so QuVINE, QProfiler and QSage were absent from the API docs entirely, along with qbiocode.evaluation.graph_evaluation. Regenerated from the real package tree: 26 pages, 128 automodule targets, covering all 128 importable modules under qbiocode/. The --separate per-module pages are no longer committed; run_apidoc regenerates them at build time with --force. Build output and deployment --------------------------- docs/Makefile sets BUILDDIR = build and CI uploads docs/build/html/, but ~250 files of rendered HTML (plus a .doctrees cache, 54 MB) were committed under docs/_build/html/ -- a second copy no tool wrote to, updated by hand. Removed from source control; both directories are gitignored. Replaced by a deploy-docs job that publishes to the gh-pages branch that ibm.github.io/QBioCode actually serves. It reuses the artifact the docs job already built (needs: docs), writes .nojekyll so Pages does not discard Sphinx's _static/, _images/, _sources/ and _modules/, and is gated on `github.event_name == 'push' && github.ref == 'refs/heads/main'` so no pull request -- including one from a fork -- can publish to the live site. peaceiris/actions-gh-pages is used rather than actions/deploy-pages because the latter needs the repository's Pages source flipped to "GitHub Actions"; pushing the branch leaves the existing setting working. The docs job's "Build documentation" step no longer sets continue-on-error, so broken docs now fail CI. The artifact upload is likewise no longer allowed to fail silently, since deploy-docs consumes it. The lint job's black/isort/mypy steps keep continue-on-error -- the tree is not fully formatted, and that is separate work. Installation docs ----------------- README.md and installation.md documented only [apps] and [all]. Both now carry the full extras matrix, and installation.md gains a QuVINE section covering why the extra is all-or-nothing, what keeps working without it, the actual QuvineDependencyError message, the setuptools<81 pin (node2vec imports pkg_resources) and the brew install cmake prerequisite for ripser on macOS. Tests ----- tests/test_docs_structure.py, 17 static checks that keep the above from regressing silently. It parses the sources and ci.yml instead of running sphinx-build, so it passes without the [docs] extra installed: every toctree entry names an existing document, no document is orphaned (honouring an explicit :orphan: / orphan: true), every automodule target is a real module, the new public modules each have a page, conf.py is anchored on __file__ and agrees with version.py, every optional QuVINE import is mocked, no rendered HTML is tracked, and the deploy job is gated and writes .nojekyll. The omegaconf mock gap was found by this test, not by review. Not verified here ----------------- No Sphinx build was run: sphinx and better_apidoc cannot be installed in this offline environment. Everything above was checked statically -- conf.py exec'd from three different working directories with a stubbed sphinx.util.logging to confirm identical results, all 128 automodule targets and 30 autosummary targets resolved against the real tree, every toctree entry and cross-reference role resolved by hand. The claim that run_apidoc's silent except was masking a real failure is a hypothesis, not a confirmed diagnosis. 149 passed, 2 skipped.
The existing tests call functions. These run the package the way a user does -- through the console scripts, a fresh interpreter, a real sphinx-build, a real QProfiler run -- because that is the only place several of this release's defects were reachable at all. The first of them justified the tier on its first run. `_resolve_scaling` unwrapped single-element sequences behind `isinstance(value, list)`, and Hydra hands the config over as `omegaconf.ListConfig`: a `Sequence`, but not a `list` subclass. Every dict-based unit test passed, and the shipped config's own `scaling: ['True']` raised `ValueError: Unrecognized scaling ['True']` on the real CLI path. The test is now on `Sequence` and the docstring names the trap. Running Sphinx for real found four broken references that reading the sources had not: a never-committed `qml_multiomics.png`, a `:ref:` in the `docname:Title` form that cannot resolve without `autosectionlabel`, a config link still pointing outside the docs tree from before the move into `qbiocode/`, and a notebook self-link using a GitHub anchor slug rather than an nbsphinx one. Plus 13 docutils warnings: `|V|` and `|E|` read as RST substitution references, and ten short section underlines. The build is verified locally now and emits no structural warnings. It is deliberately not run under `-W`: 83 docstring-formatting warnings remain across about twenty modules, and reformatting every docstring in the tree is a separate piece of work. `test_docs_build.py` asserts the structural warning classes are empty instead, which is the contract that can regress silently. Also here: link-prediction splitting no longer reseeds the caller's global RNG, `pytest` no longer needs pytest-cov to start, and `slow`/`requires_quantum` are registered and deselected by default. 322 passed, 2 skipped, 5 xfailed. The xfails are strict and carry reasons: five notebooks are genuinely truncated because they need a quantum backend, a long ensemble sweep, or anndata. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…file The `test` job installs `.[dev]`, which brings the QuVINE dependencies along, so it can never detect the one property the `[quvine]` extra exists for. The new `install-matrix` job has a bare `pip install -e .` leg and a `.[quvine]` leg; each imports the package, runs all three console scripts, and runs the suite with `-rs` so the log shows what skipped. Verified locally by making gensim, hiperwalk, node2vec, torch_geometric, community and ripser unimportable: 317 passed, 1 skipped, 5 xfailed -- the QuVINE tests skip rather than fail, and `import qbiocode` leaks none of the six. (omegaconf is deliberately not on that list: hydra-core is a base dependency and requires it, so it is present in a bare install too.) That run also found a documentation defect. Both `qbiocode/embeddings/__init__.py` and the changelog claimed `QUVINE_METHODS` is empty without the extra. It lists all 83 names, because resolving a name is stdlib-only -- so discovery and the "unknown embedding" error message work in a bare environment, and only running a method raises. The docs understated the behaviour; corrected in both places. No lock file is committed. The one inherited from the development repository was a `pip freeze` from macOS/arm64 under Python 3.12.4, presented as "the validated environment" while the matrix is three OSes x three Python versions -- wrong on eight of nine combinations, authoritative-looking on all nine. It had also outlived its own dependency set: it still pinned tensorflow==2.21.0 and keras==3.15.0, the ~600 MB dependency removed earlier in this release, and covered neither the docs tier nor scanpy/anndata/igraph/leidenalg. Copying it across would have silently undone that removal. The tiered requirements files are the single source of truth; `requirements/requirements.txt` documents the four commands that produce a lock in the environment that will use it, and the path is gitignored. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Building the docs and classifying every warning -- rather than grepping for
the ones that looked important -- turned up seven defects that the previous
pass missed. Six of them slipped through because test_docs_build.py matched
warnings by kind and none of them matched a listed kind.
Fixed in the pages:
- background.md ended with a `---` transition inside the level-3 "Video
Resources" subsection followed only by a closing paragraph. docutils
rejects that outright ("Transition must be child of <document> or
<section>"); the paragraph is a page footer, so it is now a {seealso}
admonition and needs no transition.
- installation.md linked docs/CONDA_SUBMISSION.md as ../CONDA_SUBMISSION.md.
The file exists but sits above docs/source/ and is not part of the doc set,
so MyST resolved it as an unknown source document. Now links GitHub.
- installation.md fenced a Colab cell as `python`, but `!git clone` and `%cd`
are IPython magics the Python lexer cannot tokenise. The ipython3 lexer
would handle them and ships with IPython, which is not a declared docs
dependency, so the block is fenced as `text`.
- workshops/ISMB_2025.rst indented two nested bullet lists past their parent
item's text with no blank line: docutils opened block quotes and then hit
an unexpected unindent, and neither rendered as a list.
workshops/ISMB_2026.rst had a paragraph indented one space too far.
Fixed in the API pages and config:
- napoleon_use_ivar = True. A Google-style `Attributes:` section collided
with `:undoc-members:` on every dataclass, describing each field twice --
fourteen "duplicate object description" warnings from MethodMetadata and
MethodResult alone.
- Twelve docs/source/api/*.rst pages documented a package's re-exported
names twice, once per submodule section and again in the trailing "Module
contents" block, which made the short names ambiguous ("more than one
target found for cross-reference"). That block now renders the package
docstring only, with a comment recording why so a better_apidoc
regeneration does not silently reintroduce it. Packages whose __all__ names
are not all submodule re-exports keep :members: -- qbiocode itself and
qbiocode.apps.quvine, checked programmatically rather than by eye.
Guard: test_docs_build.py now also fails on any docutils or MyST warning
reported against a file under docs/source/, since a hand-written page that
emits a markup warning does not render as intended. Docstring warnings keep
their exemption, as does myst.xref_missing for `module-*` anchors -- those
are registered through the Python domain, which MyST's local-id check cannot
see, and were confirmed present in the rendered HTML. Replaying the pre-fix
build log through the new check yields nine offenders and the post-fix log
yields none.
Build: 90 warnings -> 43, with the structural set empty. Of the 43, 33 are
docstring-formatting nits in qbiocode/ (out of scope, which is why the build
still is not run under -W) and 10 are the verified-false-positive
myst.xref_missing reports.
Suite: 326 passed, 2 skipped, 2 deselected, 5 xfailed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 fixed the train/test split but only half of reproducibility. The
integration suite caught the rest: two runs at seed 7 disagreed on one row --
embeddings=none, iteration=2, model=dt -- with accuracy 0.889 against 0.944.
Reproduced twice.
qprofiler sets np.random.seed(args['seed']) in the parent process, but model_run
fans the models out with joblib, whose loky workers are fresh interpreters
seeded from OS entropy. Every estimator left at random_state=None therefore drew
a different random state on each run. DecisionTreeClassifier permutes the
features before choosing a split, so a tie between two equally-good splits broke
either way; on a 60-sample dataset that is one whole test sample. Verified
directly: 200 fits at random_state=None give {0.889, 0.944}, at random_state=7
give {0.889}.
model_run now fills in random_state=seed for every model whose function accepts
one, deciding by signature so nb -- which has none -- is left alone rather than
crashing on an unexpected keyword. A random_state set in the config still wins.
The _opt grid-search variants had no random_state parameter at all and now take
one, threaded into both the searched estimator and the refit. compute_xgb was
missing one despite subsample=0.5, and create_xgb_model in the QPL pipeline
ignored its own seed argument while every sibling create_*_model passed it.
Seeding the global RNG inside the worker is the floor, not the mechanism: joblib
batches tasks, so how far an earlier task advanced a shared stream depends on
timing. The worker seeding is kept for randomness that has no random_state to
set -- compute_qnn's initial weights come from algorithm_globals.random, which
never crossed the process boundary either.
Tests assert the recorded parameters rather than the metrics, on purpose:
whether an unseeded estimator changes its answer depends on there being a tie to
break, so a metric comparison passes or fails by luck -- which is why the
end-to-end test caught this intermittently. The new ast guard walks every
estimator construction in the package; replayed against the pre-fix tree it
reports eight offenders, against this one none.
Eighteen `pytest.importorskip` calls guarded *first-party* modules. That is not a guard, it is an off switch: a skip and a pass are the same colour in a summary line. Injecting one `ImportError` at the top of qbiocode/utils/qutils.py turned 32 assertions across test_error_contracts.py into skips and the suite still reported green -- compute_pqk, qprofiler and gat all import that module transitively, so a single fault cascaded through four unrelated test classes. The same fault now aborts collection. Five more guarded matplotlib, networkx and pyyaml, all mandatory in requirements-base.txt, and nbformat/nbclient, which were declared nowhere at all -- so the 21 notebook tests skipped in any environment that happened not to have them, including both CI install-matrix legs. - tests/test_error_contracts.py, test_plotting_hygiene.py, test_split_reproducibility.py: import the nine first-party modules at module scope. Via `importlib.import_module`, not `from X import Y`: several of these packages re-export a function or a string under the same name as the submodule (`qbiocode.learning.compute_pqk` is both), so `from` binds whichever the parent package happens to expose at that moment -- order-dependent, and wrong for four of the nine. - test_split_reproducibility.py's guard claimed the link-prediction module "requires the [quvine] extra". It does not; it imports on a bare install. Six tests were conditional on a false premise. - nbformat/nbclient: declared in the [dev] extra and installed explicitly by both install-matrix legs, then imported directly. - tests/test_suite_hygiene.py: three new tests so none of this comes back -- no first-party module may be reached through importorskip, every guarded module must be genuinely optional (declared in an extra, not in base), and the one documented exception (the tomli shim for Python 3.10) must still be referenced by something. The three remaining importorskip calls are the legitimate ones: tomli, gensim ([quvine]) and sphinx ([docs]).
Both surfaced while executing tutorial/QProfiler/example_qprofiler.ipynb, which
has never had committed outputs.
**NMF cannot transform a leakage-free test split.** Fitting the MinMaxScaler on
train+test bounded both splits to [0, 1], so NMF.transform never saw a negative.
Fitting it on train alone -- the protocol this migration introduced -- leaves the
test split free to fall below the training minimum, and NMF is defined only on
non-negative input. Every shipped config runs `embeddings: ['pca', 'nmf',
'none']` with `scaling: ['True']`, so the leakage fix turned a silently-wrong
result into `ValueError: Negative values in data passed to X in NMF`. The test
matrix is now clipped at zero for NMF only, and the clip reports how many entries
it touched and how far out they were -- scaling the two splits together again is
the leak, and a silent clip would hide a test set outside the training range.
**folder_path did not resolve outside a directory named QBioCode.** `dir_home =
re.sub('QBioCode.*', 'QBioCode', os.getcwd())` only lands when the current
directory really sits under one literally called `QBioCode`. It does not for the
GitHub source zip (`QBioCode-main`) or a lowercase clone, and every shipped
config writes folder_path relative to the checkout root, so the tutorial failed
with `tutorial/QProfiler` doubled in the path. `_resolve_input_folder` now tries
the path as given, then the derived root, then each ancestor of the current
directory -- and the error names all three when none resolves.
Tests: 5 for the NMF clip in test_leakage_contract.py (including a guard that the
fixture really does produce out-of-range test values, or it proves nothing), and
5 for path resolution in test_error_contracts.py.
The site built successfully with 50 warnings, and a docs build that warns
successfully is a docs build nobody reads the output of. Every one of these was
a real rendering defect: docutils cannot parse the construct, so it drops or
mangles it and the published API page shows something other than what the
docstring says.
The recurring cause is a list that starts on the line immediately after its
lead-in colon. RST absorbs it into the preceding paragraph, so the bullets
render as literal "-" and "1." characters in a run-together sentence, and the
first wrapped continuation line then reads as a stray indented block quote --
which is the "Unexpected indentation" / "Block quote ends without a blank line"
pair that accounted for most of the fifty. Same fix in nine docstrings.
The rest were individual:
- random_graphs_extended: the title underline was one character short of the
title, so the module heading rendered as body text.
- link_prediction, compute_qensemble: |u - v| and |<train|test>|^2 are RST
substitution syntax. Sphinx reported an undefined substitution and dropped
the line. They are formulae, so they are inline literals now.
- hyperparameter_loader: an unindented JSON sample parsed as section titles
and definition lists. It is sample data, so it is a literal block.
- random_graphs, gcn_mf: a displayed formula and a code line indented under
prose need an explicit :: literal block, not bare indentation.
- walks/base: a numpydoc "-------" underline stranded inside a Google-style
docstring.
- ranking, dataset_evaluation, fuse: malformed Returns sections. napoleon
splits a Returns line at its first colon into type and description, so
"Dictionary with fused embeddings:" became the return *type*.
- sage: a dict-shape sketch was fenced as python, and pygments failed on the
R2 superscript. It is not Python -- <trained model> is a placeholder -- so
it is a text block.
- make_blobs: a footnote nothing cited, reported as unreferenced and rendered
as a dangling [1].
The ten myst "local id not found" warnings were false but not harmless: the
markdown links to automodule anchors do resolve (the ids are in the rendered
HTML), yet myst cannot verify them, so they were noise that a real broken link
would have hidden in. They are py:mod cross-references now, which the Python
domain does check -- so a genuinely broken one fails the build.
That leaves one warning, and it is not ours: sphinx-autodoc-typehints resolves
annotations by importing the modules they name, and pydantic's dataclass
internals annotate against _typeshed, a typing-only stub that never exists at
runtime. Exactly that subtype is suppressed in conf.py, with the reason next to
it, rather than anything broader.
With zero warnings reachable, docs/Makefile now passes -W, so any new warning
fails make html locally and in CI. tests/integration/test_docs_build.py drops
its structural-warning allowlist and its thirty-docstring exemption for a flat
assertion that the build emits nothing, plus a check that -W stays in
SPHINXOPTS -- otherwise the gate could be removed without a single test going
red.
`pip install -e .` and `pip install qbiocode` are the two things this needs to get right, so both were run for real: a scratch venv resolved all 31 base dependencies (130 packages, no conflicts), and the built wheel installed, imported from site-packages, and answered `--help` on all four console scripts. `[quvine]` resolves too, backing setuptools down to 80.10.2 to satisfy both its own `<81` pin -- node2vec imports the `pkg_resources` that 81 removed -- and qiskit-machine-learning's `>=40.1`. What that exercise found: **The sdist was 62 MB, and its size depended on shell history.** `recursive-include tutorial *.ipynb *.png *.csv *.pkl` globs the working tree, not the index. `tutorial/**/data/` and `tutorial/**/*.png` are gitignored notebook output with four fixtures force-added as exceptions, so on a checkout where the data-generation notebook had been run once the sweep packaged 239 untracked CSVs -- 111 MB of them, 123 MB unpacked. The same commit built to either 9 MB or 62 MB depending on who built it. The four fixtures are named individually now; the sdist is 9.4 MB. **`py.typed` was declared as package data and has never existed.** setuptools shipped nothing, so the bug was latent -- but the repair anyone would reach for is to create the file, and that claim would be false. `py.typed` tells a consumer's type checker the annotations here are complete and authoritative, while `disallow_untyped_defs = false` says they are not; downstream users would get confident errors derived from `Any`. The declaration is gone, with the reason recorded next to it. Both are pinned by tests/integration/test_distribution_contents.py, which builds a real sdist and wheel (`--no-isolation`, so no network) and asserts the invariant that generalizes: **the sdist contains only files git tracks.** That catches run artifacts, editor backups, a stray virtualenv and a downloaded dataset in one assertion, and needs no updating as the tree grows -- unlike a size cap. Verified to have teeth by restoring the old MANIFEST line and watching it fail on those 239 CSVs. Alongside it: the wheel carries the two config YAMLs its apps read at startup and no tests, docs or notebooks; the sdist carries the requirements file `[tool.setuptools.dynamic]` reads dependencies from at build time; and the dependency metadata is non-empty with no build-time setuptools leaked into it, which is what the old 189-line `setup.py` used to produce. `twine check` passes on both artifacts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects of the same family: each returned a plausible-looking result where it should have refused, or refused what it documents. `__init__` sliced a metadata list naming *both* `BestParams_GridSearch` and `Model_Parameters`. QProfiler writes exactly one -- `model_evaluation.py` branches on `args["grid_search"]` -- so construction from a real results table raised `KeyError: "['BestParams_GridSearch'] not in index"` with grid search off, and the mirror-image error with it on. There was no configuration in which QSage could read its own documented input, and the error named a column the user had never heard of instead of the mismatch. `predict` forwarded the caller's frame to a fitted estimator, but `train_sub_sages` appends a derived `SLGH` column after splitting. Passing exactly the columns the docstring asks for produced sklearn's "Feature names seen at fit time, yet now missing: - SLGH", blaming the caller for a column the class invents. It is now recomputed here, so a stale value cannot reach the estimator either. `predict` also took `.predict(...)[0]` internally, ranking on whichever row sorted first and discarding the rest with nothing in the output saying so. That is easy to reach by accident: complexity features are measured on the *embedded* data, so one dataset contributes a separate row per (embedding, iteration) and the obvious `results_df[features].drop_duplicates()` yields several. A multi-row input is now refused, and the message says how to get to one row. The QSage tutorial ran none of its cells; it now runs all nine against a committed 15-dataset QProfiler table, predicts per embedding, and asserts a save/reload round-trip reproduces the whole prediction table. Its picks are mostly wrong, and the notebook says so rather than tuning the fixture until the demo flatters the method -- 15 datasets demonstrate the mechanism, not a fitted surrogate, so the text points the reader at `r2` first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three log lines sat above `_validate_config` and indexed `args['n_jobs']`, `args['model']` and `args['backend']` directly. A config missing any of them died with a bare KeyError raised from inside a logging statement -- the exact "error attributed to the wrong thing" that `_validate_config` exists to replace, and it named one missing key where the validator names all of them at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Committed with 0 of 8 code cells run. `nbsphinx_execute = 'never'`, so it published as a page of empty cells. No source changed -- it needed running, not fixing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`test_the_sdist_ships_only_files_git_tracks` failed with 231 strays, and
MANIFEST.in was not the cause: building the same commit from a clean copy gave
285 members and 0 strays. setuptools *unions* the previous
`qbiocode.egg-info/SOURCES.txt` into each new sdist rather than recomputing it,
so a checkout that ever built with a broader MANIFEST.in keeps shipping files
MANIFEST.in no longer names -- here, 231 CSVs of notebook run output. A release
built locally would have published them.
The fixture now builds from a staged copy of the working tree with `*.egg-info`
and build residue excluded. Deliberately not a clean `git archive`: MANIFEST.in
globs the *working tree*, so run output under `tutorial/**/data/` has to stay
visible or the test proves nothing. That is a second, independent
builder-dependence channel, and the stale-metadata one was masking it
entirely. CONTRIBUTING now documents the `rm -rf dist build *.egg-info` this
implies for local builds; CI is immune by construction, since a fresh
`actions/checkout` has no metadata to be stale.
`test_every_guarded_module_is_actually_optional` failed on the same commit
because that test's own `importorskip('build')` claimed the `[dev]` extra
provides `build`, which was untrue -- so the guard could only ever skip. `build`
is now in `[dev]`, and the hygiene rule reads pyproject's extras as well as the
tiered requirements files: `quvine` and `docs` are mirrored between the two,
but `dev` and `apps` exist only as extras, so reading just the files made a
genuinely optional dependency look undeclared.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xgboost and torch each vendor their own copy of libomp under the same
install name, so importing both maps two independent LLVM OpenMP runtimes
into one process. The first to initialise claims the process-wide runtime
state, and the second dies when it opens a parallel region:
Segmentation fault: 11
EXC_BAD_ACCESS (KERN_INVALID_ADDRESS at 0x580)
libomp.dylib __kmp_suspend_initialize_thread
libomp.dylib __kmp_launch_worker
qbiocode forced the losing order. qbiocode/__init__.py imports .embeddings
before .learning, and .embeddings eagerly imported ConvAutoencoder, whose
first line is `import torch` -- so torch's runtime was installed first in
every process that imported the package, and every XGBoost fit that
followed was a segfault. That took out QPL's qpl_xgb arm, compute_xgb, and
so any QProfiler run configured with an XGBoost model. There is no Python
traceback and nothing to catch, so it surfaced only as DeadKernelError
from a notebook and a silent exit 139 from a script, and it looked like a
failure in the quantum step because the other four QPL learners had
already printed.
Measured, fitting an XGBClassifier: xgboost alone is fine, xgboost then
torch is fine, torch then xgboost segfaults. Turning parallelism down does
not help -- neither n_jobs=1 on the estimator nor on the surrounding
search -- because the fault is inside the OpenMP runtime, below joblib.
Only the import order or disabling OpenMP outright avoids it.
So: ConvAutoencoder resolves through a lazy module __getattr__, and
`import qbiocode` no longer loads torch at all. Nothing in the package
uses that class, so the eager import bought nothing. And the package body
now calls preload_openmp_libraries() before any submodule import, which
initialises xgboost's runtime first and keeps a caller's later
`import torch` safe; it warns if torch was already imported first, since
by then no ordering can help.
qbiocode/utils/_openmp.py records the measurements, because the next
person to see this will have a bare SIGSEGV and no other evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All seven tuned learners took one keyword per tunable hyperparameter, each
defaulting to [], and handed all of them to GridSearchCV unconditionally.
Any config naming a subset therefore died inside sklearn on whichever
parameter it had left alone:
ValueError: Parameter grid for parameter 'colsample_bytree' need to be
a non-empty sequence, got: []
The message names a parameter the user never wrote and points at sklearn
rather than at the config, which is the opposite of the useful direction.
It also made a deliberately small grid inexpressible: trimming a demo
config down to two parameters was indistinguishable from corrupting it,
which is why the QPL tutorial shipped a 2430-combination XGBoost grid
costing over six hours.
build_param_grid keeps only the values actually supplied, leaving every
unmentioned hyperparameter at the estimator's own default -- which is what
"not tuned" should mean. When nothing at all was supplied it raises a
message naming the config block to add and the grid_search: False opt-out,
rather than blaming an arbitrary parameter.
Two related fixes came with it. A bare string is wrapped rather than
searched character by character: `max_features: sqrt` in YAML was
previously searched as ['s', 'q', 'r', 't'], four invalid values that
produced no error and a meaningless best_params_. And the [] defaults are
now None -- a shared mutable default is a hazard whether or not this code
happened to mutate it.
compute_xgb_opt also now reports a bootstrap grid instead of silently
doubling. XGBoost has no bootstrap parameter, but its sklearn wrapper
accepts unknown keyword arguments without complaint, so the shipped
tutorial config was never an error; it just searched twice as many
combinations, every duplicate returning the same model.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The notebook could not be run. Measured on a laptop, its two compute cells cost roughly nine hours: the quantum cell about 2.8h (32 requested datasets at 10-15 features, and PQK simulation cost is exponential in the qubit count, which equals the feature count under embeddings: ['none'] -- 6 qubits 6.3s per 100 samples, 15 qubits 138.3s), and the classical cell about 6.4h (a 2430-combination grid x 5-fold CV, 265s per dataset and iteration). Cut to 4 datasets at 6-8 features and a 16-combination grid, the whole notebook now executes in 270s and shows the same behaviour. The sizing comment in the dataset cell carries the measurements so the next person widening it knows what it costs. Three things were also simply wrong. It invoked compute_pqk while its own overview described compute_qpl. PQK fits a single SVC and reports one row named 'pqk'; QPL projects through the same feature map and fits five classical learners, reporting qpl_svc, qpl_rf, qpl_xgb, qpl_mlp and qpl_lr. The five-learner comparison is what the notebook is about, so the config was the thing that was wrong. A qpl_args block had to come with it: model_run reads args[method + "_args"], so a pqk_args block is invisible to model: ['qpl'] and QPL silently fell back to its signature defaults while the config appeared to ask otherwise. The quantum-advantage cell hard-coded qml_models as ['pqk_lr', 'pqk_svc', 'pqk_rf', 'pqk_mlp', 'pqk_xgb'] -- the five QPL learner names wearing the PQK prefix, a set neither implementation can produce. Nothing ever matched, so the win rate printed 0.0% however well the quantum models did. It now reads the names out of the results by prefix, and reports 50.0%. rf.yaml was fully configured and loaded by nothing, which is why nobody noticed its folder_path pointed at output from a different notebook -- gitignored, committed nowhere, so from a fresh clone the run died on a folder that had never existed. It is now the second tuned classical baseline, alongside XGBoost, which is what makes the quantum comparison worth making. Smaller: the notebook is idempotent now (ModelResults.csv is appended to, so a second pass silently doubled every row), and the plotting cell's shared output_dir/tag moved out of an `else:` branch that left the next cell raising NameError on exactly the run where you want the message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The notebook was 9 of 10 code cells executed, and nbsphinx_execute is 'never', so it published as a partly-empty page. It needed running, not fixing: all 10 cells now execute end to end with no source changes. Its KNOWN_TRUNCATED entry claimed "its import cell needs anndata, which is not installable in the environment this test suite was written in". anndata 0.13.3 and scanpy 1.12.4 are both installed and both import fine, so the reason was false and the strict xfail was hiding a notebook that only ever needed to be run. Also fixes a reversed mirror direction: the tutorial/ copy of QEnsemble_example_blobs.ipynb said it was "mirrored under tutorial/", pointing at itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sc_binary_quvine_2x2_qprofiler.ipynb had 7 of 8 code cells run in both trees. The unexecuted cell was the notebook's conclusion -- the 2x2 comparison of classical vs QuVINE embeddings against classical vs quantum models -- so with nbsphinx_execute = 'never' the published page ended just before its result. Run: 8/8 cells, 0 errors, 3 figures, 228s, 36 result rows (4 embeddings x 3 models x 3 iterations). Both copies carry identical outputs. Its two KNOWN_TRUNCATED entries in test_notebook_execution.py are removed, so test_no_notebook_is_half_executed now enforces the completed state rather than recording the truncation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The notebook had 0 of 7 code cells executed, so its docs page published the source with nothing under it, and it could not simply be run to fix that: - folder_path was 'tutorial/QProfiler/data/ld_data', a repository-relative path. _resolve_input_folder searches every ancestor of the cwd, so from the docs copy it resolved to the *tutorial* tree's data -- both copies read one input -- and from an installed package it resolved to nothing. It is now 'data/ld_data', the directory the notebook's own first cell generates next to itself, which is correct from either tree. - The two configs/config.yaml copies had drifted: different model lists, embeddings, n_jobs and credential comments, so the same notebook behaved differently depending on which tree you opened. They are now byte-identical and sized for a tutorial (embeddings ['none','pca'], 8 models, n_jobs 1, iter 2), with the full ten-model list kept as a comment for a real sweep. - xgb_args and gridsearch_xgb_args were absent, so xgb ran on _model_args' empty-dict fallback with nothing documenting what it can tune. Both blocks are now present. - Replaced ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha='right') with ax.tick_params(axis='x', rotation=45). Setting tick labels on an axis whose tick locations are not fixed warns in matplotlib >= 3.5 and misattaches the labels if the locator picks a different tick count. Run: 7/7 cells, 0 errors, 7 figures, 80s, in a clean sandbox from empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six notebooks were re-executed earlier in this release with MPLBACKEND=Agg in the environment. Inside an ipykernel the backend that produces inline output is module://matplotlib_inline.backend_inline, which captures plt.show() as an image/png display output; Agg makes plt.show() a silent no-op. The cells ran, reported success, and emitted no figure at all -- and because nbsphinx_execute = 'never', the published pages rendered code and printed text with every plot gone. test_no_notebook_is_half_executed stayed green throughout, since the cells really were executed. Caught by diffing image/png counts against the pre-existing commits: quvine_sc_cd4_vs_cd8.ipynb had 4 figures at c19b5ac^ and 0 after. Re-run under the default inline backend, figures intact: QPL_example.ipynb 4 figures (both trees) qsage.ipynb 7 figures (both trees) quvine_sc_t_vs_mono.ipynb 2 figures quvine_sc_cd4_vs_cd8.ipynb 4 figures Every git-tracked notebook was audited against origin/main to bound the damage to these six. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
embed.pqk's data_map_func ended in `return float(coeff)`. Qiskit calls a
data_map_func with symbolic parameters while it constructs the circuit --
PauliFeatureMap.pauli_block does
params = ParameterVector("_", length=len(pauli_string))
time = self._data_map_func(np.asarray(params))
-- so float() raised "Parameter expression with unbound parameters {...} is not
numeric" before anything executed. data_map=True is the default, making pqk()
unusable for Z, ZZ and P alike.
It went unnoticed because compute_pqk held a second, already-fixed copy of the
same function: the QProfiler `pqk` model worked while calling pqk() directly --
what tutorial/PQK - OV.ipynb does -- did not.
Both copies now call one shared qutils.unit_coefficient_data_map, which returns
a float for numeric input and the unevaluated ParameterExpression for symbolic
input, so the two PQK paths cannot diverge again. The now-unused
`from functools import reduce` import is dropped from both modules.
Also replaces the bare KeyError from get_backend_session's unchecked
args["seed"] / args["shots"] with a ValueError naming the missing keys, what
each is for, and the keys that were supplied -- this is how the notebook's
failure first presented once the data map worked.
tests/test_feature_map_data_map.py: numeric contract, symbolic input at three
vector lengths, all four (encoding, entanglement) pairs built through
get_feature_map, a pqk() round trip on the statevector simulator, and the
missing-seed message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
QEnsemble_example_blobs.ipynb had 10 of 15 code cells executed in both trees.
The unrun cells were the xgb, qcosine, qensemble and qensemble_random_unitary
arms plus the post-processing that compares them -- so the published page
stopped immediately before the comparison the tutorial exists to make.
Two defects kept it there:
- Every arm is guarded against experiments/predictions.pkl with
`method not in predictions[dataset_name].keys()`. A method that ran but
produced nothing is also a key, so the committed cache's empty xgb_gs frame
(0 rows, recorded in an environment without XGBoost) counted as done and was
never retried. The notebook printed "XGBoost grid search results are empty
... Skipping XGBoost" on an install where xgboost is in requirements-base and
importable. The guard is now needs_run(predictions, dataset_name, method,
rerun), a helper_functions.py addition that treats an empty cached result as
absent.
- helper_functions.py had re.sub('\ ', '_', metric) -- an invalid escape
sequence, a SyntaxWarning today and a SyntaxError in a future Python. It is
now re.sub(' ', '_', metric), the same regex.
Run: 15/15 cells, 0 errors, 3 figures, 41s. xgb_gs fits its 486 candidates over
3 folds and xgb its 90 rows, and the post-processing now reports XGBoost against
every quantum arm with significance tests. predictions.pkl and the three
Blob_max_median_*.pdf figures carry the completed comparison. The cheap arms
recompute on a fresh run; the quantum arms still come from the cache, which is
what makes the notebook openable without a long sweep.
Both trees' notebooks and helper_functions.py are byte-identical.
With this, KNOWN_TRUNCATED in test_notebook_execution.py is empty: every
notebook in the tree is either a clean template or fully executed. The mechanism
stays so a future truncation is recorded deliberately rather than by weakening
test_no_notebook_is_half_executed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rials
The term names a complexity-theoretic result. What these notebooks measure is a
per-dataset score difference between quantum and classical arms on small
simulated data, which is not evidence of one, so the headings, labels, prints
and figure names now say what is actually being compared.
QPL_example.ipynb: section 9 becomes "Quantum vs. Classical Comparison", its
figure becomes {tag}_quantum_vs_classical.png, and its prints name the datasets
a quantum model ranked first on. Its committed output text is patched to match:
every changed byte is a print literal, so no number moved and re-running a
40-minute simulation would only have perturbed results that are not at issue.
QEnsemble_example_blobs.ipynb: the "Quantum Advantage" takeaway becomes
"Quantum Ensembling" -- superposition over classifiers is a mechanism, not a
demonstrated advantage. Its two "theoretical quantum advantage" mentions in the
further-work lists are kept, where the term is used correctly.
quvine_sc_cd4_vs_cd8.ipynb: the quantum_adv column, comments and plot axis
become "quantum recall gain", which is the quantity -- recall(ctqw,dtqw) −
recall(node2vec). This one had to be re-executed because its axis label is
rasterized into a committed figure, and the re-run moved its numbers materially
(one stratum p=0.005 -> p=0.376, several bootstrap correlations flipped sign):
quvine/embedding/word2vec.py trains gensim Word2Vec with workers=8 and takes no
seed, so no SGNS embedding is reproducible however base_seed is threaded through
the walks. Recorded in the CHANGELOG rather than fixed here -- workers=1 plus a
seed is an 8x slowdown that would change every committed QuVINE number.
background.md keeps its discussion of the concept and the Huang et al. title,
and evaluate_graph keeps compute_quantum_advantage_metrics and its
quantum_advantage_* keys: those are public API and saved-CSV column names, not
worth breaking over wording.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
run_model returned best_model.score(x_test, y_test), which for an SVC is
accuracy. It was then named f1_score, printed as "Test F1 score", stored in
F1_Quantum/F1_Classical, written to PQK_OV_results.csv and used as the plot's
"F1 Score" axis -- and it disagreed with the GridSearchCV in the same function,
which selects on 'f1_weighted'. The tell in the previous run was that all six
reported values were exact k/59 fractions of the 59-sample test set. Score the
held-out set with f1_score(..., average='weighted') so the number reported and
the criterion selected on are the same metric.
Also print('\n=' * 60) -> print('\n' + '=' * 60): the original repeats
newline-plus-equals sixty times, so five section breaks each rendered as 60
lines containing a single '='.
Outputs land in a follow-up commit; the corrected metric requires re-executing
the grid search.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
param_grid was one flat dict sweeping gamma across all four kernels, but SVC ignores gamma when kernel='linear'. That made 35 C x 37 gamma = 1,295 fits of just 35 distinct linear models -- 24.3% of the 5,180-combination grid spent on exact duplicates -- and left a linear winner reporting a best gamma that the model never read. Use a list of grids so gamma applies only to poly/rbf/sigmoid: identical search space, 3,920 combinations instead of 5,180. Selection is unaffected beyond no longer reporting a meaningless gamma for a linear best model. The config cell now prints the combination count, since this grid is the notebook's entire runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Re-executed `PQK - OV.ipynb` end to end (8/8 cells, 1 figure, 0 errors) on top of the three source fixes in e527328 and e2a7b12, and committed the outputs to both the tutorial/ and docs/source/tutorials/ copies (they are byte-identical). The numbers moved, because the previous outputs were not what they claimed to be. `best_model.score()` returns accuracy for SVC, so every value in the F1_ columns, the summary CSV and the plot was an accuracy -- all six were exact k/59 fractions of the 59-row test set, which is also why two modalities reported identical quantum and classical values. Post-fix every value is a genuine weighted F1 (non-integer k/59, matching the scoring='f1_weighted' the grid search selects on): F1_Quantum F1_Classical Improvement_% mirna 0.526242 0.591067 -10.967430 methy 0.403709 0.637668 -36.689845 exp 0.519209 0.526242 -1.336443 integrated 0.560082 0.625145 -10.407632 Average quantum 0.5023 vs classical 0.5950 (-14.85%). The result is negative: on this dataset the PQK arm does not beat the classical SVM. The notebook's prose never claimed otherwise ("Evaluated quantum-enhanced vs. classical SVM"), so no text needed changing. `Grid combinations: 3920` confirms the deduplicated grid, and cells 4 and 6 now emit 6 and 8 full rules with no lone '=' lines. Also add the .gitignore rules this notebook needed and never had: it downloads a 46 MB archive and extracts ~250 MB of TCGA tables beside itself, then writes a results CSV there. Without `tutorial/**/OV/`, `tutorial/**/ovarian.zip` and `tutorial/**/PQK_OV_results.csv` a reader's first run leaves ~300 MB of untracked files that `git add -A` would commit. Every byte is reproducible from the URL in the notebook's own download cell. Added `tutorial/**/qpl_projections/` for the same reason -- its sibling `pqk_projections/` was already ignored, this one was missed.
The notebook died partway through cell 3 with no traceback -- a bare dead
kernel. Three crash reports on this machine, all identical:
EXC_BAD_ACCESS SIGSEGV / Segmentation fault: 11
libomp.dylib __kmp_suspend_64<false, true>
libomp.dylib __kmp_hyper_barrier_release
libomp.dylib __kmp_fork_barrier
libomp.dylib __kmp_launch_worker
libomp copies mapped: 3
Three separate LLVM OpenMP runtimes end up in one process: torch's, the one
qiskit-aer vendors, and the interpreter's own (this venv is built on
Anaconda). Whichever initialises second finds bookkeeping it did not create
and dies the first time it opens a parallel region. Isolated per method,
the three torch-backed entries in the notebook's `methods` list each
reproduce it standalone:
quvine_rwr, filter_rwr_heat, node2vec, netmf ok
appnp, gat_rwr_poly, graphgps_rwr_heat exit 139 (SIGSEGV)
It is not a Jupyter or VS Code problem -- nbclient dies the same way.
Tested fixes: OMP_NUM_THREADS=1 works, importing torch before qbiocode
works, KMP_DUPLICATE_LIB_OK=TRUE does not.
The fix is one line the sibling notebook already had.
`quvine_sc_t_vs_mono.ipynb` sets OMP_NUM_THREADS=1 in its imports cell;
`example_quvine.ipynb` had zero occurrences of it, and it is the one that
calls the torch methods. Set it before the qbiocode import, since the
runtime initialises at import time.
Re-executed end to end: 11/11 cells, 4 figures, 0 errors, 248s. Cell 3
wraps each embed in a try/except that records NaN, so completion alone
proves nothing -- verified 180 rows collected, no NaN in the summary, and
the error key never populated, so appnp (0.698), gat_rwr_poly (0.862),
gat_ctqw_poly (0.863), graphgps_rwr_heat (0.863) and graphgps_ctqw_heat
(0.873) all really ran.
CHANGELOG also records why this cannot be fixed by reordering instead:
preload_openmp_libraries() imports xgboost at `import qbiocode` so
xgboost's runtime wins, which is what the QPL pipelines need and is exactly
why torch loses here. The orderings are mutually exclusive, so
OMP_NUM_THREADS=1 is the only setting that satisfies both.
_resolve_input_folder tried dir_home / folder_path -- a root frozen at import time -- before walking the ancestors of the current directory. A CLI run never notices, because cwd has not moved since import. A notebook kernel, a test session, or a batch driver that changes directory does: the frozen root wins, and a run launched from QBioCode-main/tutorial/QProfiler silently reads a different checkout's CSVs and reports them as its own. Candidate order is now: the path as given, each ancestor of the current directory, a root derived from the current directory, and dir_home last -- still there so existing configs resolve, no longer able to shadow the caller. Also marks dir_home :meta private:. autodoc renders module-level values, so api/qbiocode.apps.qprofiler.html was publishing the doc builder's own absolute filesystem path to GitHub Pages.
Seventeen notebooks stored the running machine's own filesystem paths in their output: a node2vec pkg_resources warning, tqdm's 'IProgress not found' banner, and print() echoes of the working and data directories. nbsphinx_execute is 'never', so those strings publish verbatim to GitHub Pages -- where they name a directory that does not exist on the reader's machine and disclose the author's home directory layout for no benefit. All of them are now <env>/ or <repo>/ placeholders. Verified text-only: cell counts, execution_counts, output counts and figure payload sizes are byte- identical to HEAD in all seventeen files, so no metric, table or plot moved. archive/tutorial_notebooks/analyses/visualize_results.ipynb keeps its five -- they are commented-out source paths recording where an earlier analysis ran, not stray output, and that tree is not part of the published site.
Both notebooks existed only under tutorial/, so neither reached the site.
Copy them into docs/source/tutorials/QuVINE/ and give each a section in
tutorials.md §4, which now presents all four QuVINE notebooks in reading
order (synthetic-graph API walkthrough, CD4 vs. CD8, T vs. monocyte, then
QuVINE-through-QProfiler) with What You'll Learn / Key Concepts blocks and
toctree entries.
example_quvine.ipynb 11 cells, 4 figures -- 12 methods x 3 SBMs
x 5 iterations, then complexity vs. macro-F1
quvine_sc_t_vs_mono.ipynb 8 cells, 2 figures -- seed->eval node
classification and seed->target ranking on
the 800-cell two-view graph
Publishing example_quvine.ipynb surfaced 127 KB of dead metadata.widgets
(165 tqdm records with no widget output to render -- every progress bar was
already captured as plain text). nbsphinx warned "nbsphinx_widgets_path not
given and ipywidgets module unavailable", fatal under sphinx-build -W, and
the payload would have shipped to Pages unrendered. Removed from both
copies; nothing else changed (same 25 cells, same execution counts, outputs
byte for byte). Adding ipywidgets to [docs] would have silenced the warning
and kept the dead payload.
Two static guards in tests/test_docs_structure.py (19 checks, up from 17):
metadata.widgets with no widget output, and any /Users, /home or
/private/tmp string in a notebook under tutorial/ or docs/source/tutorials/
-- so neither condition can silently return.
Verified: sphinx-build -W --keep-going -a -E exits 0 with no warnings; both
pages render with their figures and outputs (109 KB and 105 KB); every
tutorials.html link resolves; no absolute path anywhere in the built site.
Separate from the migration commits: additive, independently revertable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Type of Change
Related Issue(s)
Fixes #
Closes #
Related to #
Changes Made
Component(s) Affected
Testing
Test Configuration
Tests Performed
Test Details
# Example test code or commands usedScreenshots/Output (if applicable)
Checklist
Code Quality
Documentation
Testing
Dependencies
requirements.txtorsetup.pyif neededBreaking Changes
Breaking Changes Description
Additional Notes
Reviewer Notes
For Maintainers