fix(nlp): pin spaCy model as locked dependency; one-step contributor setup#182
Merged
Merged
Conversation
…setup The spaCy model (en_core_web_md) kept "disappearing," requiring 10+ manual reinstalls over months. Root cause: it was installed out-of-band via `python -m spacy download`, leaving it untracked in uv.lock. uv's exact sync (triggered by any `uv run`/`uv sync`, including `make ready` and CI) then removed it as an extraneous package. The NLP layer silently fell back to the string/heuristic layers, masking the deletion. Fix: pin the model wheel URL as a real dependency so `uv sync` installs and preserves it everywhere (local venv, docker compose, CI, multi-arch publish). The wheel is py3-none-any, so one artifact serves all platforms. - pyproject.toml/uv.lock: pin en_core_web_md (>=3.8,<3.9), locked with hash - Dockerfile: drop the now-redundant `spacy download` step - Makefile: add `make setup` (one-step, idempotent env provision + verify) and `make verify-env`; run verify-env as the first `ready` gate step - scripts/verify_dev_env.py: assert the model actually loads (loud, not silent) - tests/unit/test_dev_env_spacy_model.py: CI guard reusing check_spacy_model() - docs/README: replace manual `spacy download` with `make setup` Verified with `make ready`: 521 passed, parity 4/4, docs validation passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The spaCy NLP model (
en_core_web_md) kept "disappearing," requiring 10+ manual reinstalls over several months.Root cause
The model was installed out-of-band via
python -m spacy download, which leaves it untracked inuv.lock. uv's sync is exact by default — it removes any package not in the lockfile. Everyuv run/uv sync(includingmake ready, localmaketargets, and CI) reconciled.venvto the lockfile and deleted the untracked model. The NLP layer then silently fell back to the string/heuristic layers, which is why the failure looked random rather than like a one-time setup miss.Fix
Pin the model wheel URL as a real dependency so
uv syncinstalls and preserves it in every environment — local venv (make),docker compose, CI, and the multi-arch Docker Hub publish. The wheel ispy3-none-any(arch-independent), so a single artifact serves bothlinux/amd64andlinux/arm64; verified the pinned URL resolves and the model declaresspacy_version >=3.8,<3.9(compatible with locked spaCy 3.8.x).Changes
pyproject.toml/uv.lock— pinen_core_web_mdwheel URL; locked with sha256. Confirmed it now survives an exactuv sync.Dockerfile— drop the now-redundantpython -m spacy downloadstep (theuv sync --frozensteps install it).Makefile— addmake setup(one-step, idempotent: guards onuv, syncs, verifies) andmake verify-env; runverify-envas the firstreadygate step (pure verification, honoring the gate's no-mutation contract).scripts/verify_dev_env.py— assert the model actually loads, turning the old silent degradation into a loud, actionable failure.tests/unit/test_dev_env_spacy_model.py— CI guard (unitlane) reusingcheck_spacy_model()so setup,make ready, and CI share one source of truth.README.md/ docs — replace manualspacy downloadinstructions withmake setup; add "do not install out-of-band" warnings.Contributor experience
New contributors now run a single command:
Trade-off
The model URL pins an exact version decoupled from spaCy's floating version. If spaCy ever resolves to
3.9.x, the pinned3.8.0model becomes incompatible — but that now surfaces as a visibleuv lockdiff / loud load failure (caught by the CI guard) instead of silent degradation.Verification
make ready— all 6 gates green: 521 passed, parity 4/4, docs validation passed.🤖 Generated with Claude Code