Skip to content

fix(nlp): pin spaCy model as locked dependency; one-step contributor setup#182

Merged
touchthesun merged 1 commit into
mainfrom
fix/spacy-model-locked-dependency
Jul 2, 2026
Merged

fix(nlp): pin spaCy model as locked dependency; one-step contributor setup#182
touchthesun merged 1 commit into
mainfrom
fix/spacy-model-locked-dependency

Conversation

@touchthesun

Copy link
Copy Markdown
Contributor

Problem

The spaCy NLP model (en_core_web_md) kept "disappearing," requiring 10+ manual reinstalls over several months.

Root cause

The model was installed out-of-band via python -m spacy download, which leaves it untracked in uv.lock. uv's sync is exact by default — it removes any package not in the lockfile. Every uv run/uv sync (including make ready, local make targets, and CI) reconciled .venv to the lockfile and deleted the untracked model. The NLP layer then silently fell back to the string/heuristic layers, which is why the failure looked random rather than like a one-time setup miss.

Fix

Pin the model wheel URL as a real dependency so uv sync installs and preserves it in every environment — local venv (make), docker compose, CI, and the multi-arch Docker Hub publish. The wheel is py3-none-any (arch-independent), so a single artifact serves both linux/amd64 and linux/arm64; verified the pinned URL resolves and the model declares spacy_version >=3.8,<3.9 (compatible with locked spaCy 3.8.x).

Changes

  • pyproject.toml / uv.lock — pin en_core_web_md wheel URL; locked with sha256. Confirmed it now survives an exact uv sync.
  • Dockerfile — drop the now-redundant python -m spacy download step (the uv sync --frozen steps install it).
  • Makefile — add make setup (one-step, idempotent: guards on uv, syncs, verifies) and make verify-env; run verify-env as the first ready gate step (pure verification, honoring the gate's no-mutation contract).
  • scripts/verify_dev_env.py — assert the model actually loads, turning the old silent degradation into a loud, actionable failure.
  • tests/unit/test_dev_env_spacy_model.py — CI guard (unit lane) reusing check_spacy_model() so setup, make ready, and CI share one source of truth.
  • README.md / docs — replace manual spacy download instructions with make setup; add "do not install out-of-band" warnings.

Contributor experience

New contributors now run a single command:

make setup

Trade-off

The model URL pins an exact version decoupled from spaCy's floating version. If spaCy ever resolves to 3.9.x, the pinned 3.8.0 model becomes incompatible — but that now surfaces as a visible uv lock diff / loud load failure (caught by the CI guard) instead of silent degradation.

Verification

make ready — all 6 gates green: 521 passed, parity 4/4, docs validation passed.

🤖 Generated with Claude Code

…setup

The spaCy model (en_core_web_md) kept "disappearing," requiring 10+ manual
reinstalls over months. Root cause: it was installed out-of-band via
`python -m spacy download`, leaving it untracked in uv.lock. uv's exact sync
(triggered by any `uv run`/`uv sync`, including `make ready` and CI) then
removed it as an extraneous package. The NLP layer silently fell back to the
string/heuristic layers, masking the deletion.

Fix: pin the model wheel URL as a real dependency so `uv sync` installs and
preserves it everywhere (local venv, docker compose, CI, multi-arch publish).
The wheel is py3-none-any, so one artifact serves all platforms.

- pyproject.toml/uv.lock: pin en_core_web_md (>=3.8,<3.9), locked with hash
- Dockerfile: drop the now-redundant `spacy download` step
- Makefile: add `make setup` (one-step, idempotent env provision + verify) and
  `make verify-env`; run verify-env as the first `ready` gate step
- scripts/verify_dev_env.py: assert the model actually loads (loud, not silent)
- tests/unit/test_dev_env_spacy_model.py: CI guard reusing check_spacy_model()
- docs/README: replace manual `spacy download` with `make setup`

Verified with `make ready`: 521 passed, parity 4/4, docs validation passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@touchthesun touchthesun merged commit e304013 into main Jul 2, 2026
5 checks passed
@touchthesun touchthesun deleted the fix/spacy-model-locked-dependency branch July 2, 2026 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants