Skip to content

fix(deps): make h5py a core dependency; BrainCollection.fit() requires it - #469

Merged
ejolly merged 1 commit into
masterfrom
fix/braincollection-h5py-dependency
Jul 27, 2026
Merged

fix(deps): make h5py a core dependency; BrainCollection.fit() requires it#469
ejolly merged 1 commit into
masterfrom
fix/braincollection-h5py-dependency

Conversation

@ljchang

@ljchang ljchang commented Jul 27, 2026

Copy link
Copy Markdown
Member

Problem

BrainCollection.fit() always writes an HDF5 fit bundle, and
nltools/data/collection/execution.py imports h5py unguarded:

def _write_bundle(...):
    import h5py                      # no try/except, no extras check
    with h5py.File(tmp, "w", locking=False) as f:

But h5py was declared only in the optional h5 extra. Installing plain
nltools and calling BrainCollection.fit() fails with:

nltools.data.collection.execution.BrainCollectionWorkerError:
  [idx=4] ModuleNotFoundError: No module named 'h5py'

Because it is raised inside a joblib worker, the traceback is ~20 frames of
joblib/parallel.py before the real cause, which makes it hard to diagnose —
especially for students, which is how I found it (porting the
dartbrains course to 0.6).

Two things hid this:

  • The dev dependency group installs h5py, so it never reproduces in development.
  • nltools/io/h5.py does guard its import and raise an actionable error, but
    only for the BrainData .h5 path. The collection path never goes through it.

Fix

Move h5py to [project].dependencies. BrainCollection is a headline v0.6
feature and is unusable without it, so it is not an optional add-on.

hdf5plugin deliberately stays in the h5 extra: it only registers
blosc/zstd/lz4 filters for reading compressed legacy files, the bundle writer
uses no compression filters, and nltools/io/h5.py already guards it. The
h5 extra keeps listing both so existing nltools[h5] pins resolve unchanged.

Regression guard

Adds nltools/tests/support/test_packaging.py, which parses pyproject.toml
and asserts that modules imported unconditionally by the collection execution
path are declared as core dependencies. Written red-first — it fails on
master with:

AssertionError: nltools/data/collection/execution.py imports ['h5py']
unconditionally, but they are not in [project].dependencies.

The check walks the AST and ignores imports inside try/except, so guarding
an import (with a friendly message) is still a valid way to satisfy it.

Docs

The migration guide still described BrainCollection as "not yet available
(scaffold)"
in three places and pointed at nltools/data/collection/SPEC.md,
deleted in 378c0c1. The class is fully functional — I verified
from_paths.smooth().fit(model='glm', X=...).compute_contrasts()
.ttest() / .permutation_test() / .predict(spatial_scale=...) end to end.

Replaced with a worked example plus the two contracts that aren't obvious from
the signatures:

  • the X= callable receives a _DesignContext, not a DesignMatrix
  • from_paths(design_paths=...) passes paths through unparsed, so the builder
    must construct the DesignMatrix (matching the source comment: "DesignMatrix
    has no read() classmethod yet"
    )

Testing

  • uv run poe lint — clean
  • nltools/tests/support/ — 177 passed
  • nltools/tests/data/collection — 276 passed

(Both suites need HF_HUB_DISABLE_IMPLICIT_TOKEN=1 on my machine; a stale local
HF token 401s against the public nltools/niftis dataset. Unrelated to this change.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01YVDnsKutg6Yqv99dYmUeuP

…s it

`nltools/data/collection/execution.py` imports h5py unguarded — every
`BrainCollection.fit()` writes an HDF5 fit bundle — but h5py was declared only
in the optional `h5` extra. Anyone installing plain `nltools` hit

    ModuleNotFoundError: No module named 'h5py'

raised from inside a joblib worker, which surfaces as a
BrainCollectionWorkerError with a stack that points at joblib rather than at
the missing dependency.

This is invisible in development because the dev group installs h5py, and
invisible to the `h5` extra's own guard: `nltools/io/h5.py` raises a friendly
error for the BrainData `.h5` path, but the collection path never goes through
it.

hdf5plugin stays optional — it only registers blosc/zstd/lz4 filters for
reading compressed legacy files, the bundle writer uses no compression
filters, and `nltools/io/h5.py` already guards that import.

Adds a packaging test that parses pyproject.toml and asserts every module
imported unconditionally by the collection execution path is declared in
[project].dependencies, so the class of bug can't recur silently.

Also refreshes the migration guide, which still described BrainCollection as
"not yet available (scaffold)" and pointed at the deleted
nltools/data/collection/SPEC.md. Replaced with a worked example and the two
contracts that aren't obvious from the signature: the `X=` callable receives a
`_DesignContext` rather than a DesignMatrix, and `from_paths(design_paths=)`
passes paths through unparsed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YVDnsKutg6Yqv99dYmUeuP
@ejolly
ejolly force-pushed the fix/braincollection-h5py-dependency branch from 82f7fff to 0578932 Compare July 27, 2026 20:29
@ejolly
ejolly merged commit bb22020 into master Jul 27, 2026
5 checks passed
@ejolly
ejolly deleted the fix/braincollection-h5py-dependency branch July 27, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants