Skip to content

fix: mount the model registry from MODEL_CONFIG, not the git checkout#36

Open
JasonWildMe wants to merge 1 commit into
mainfrom
fix/model-config-mount-outside-checkout
Open

fix: mount the model registry from MODEL_CONFIG, not the git checkout#36
JasonWildMe wants to merge 1 commit into
mainfrom
fix/model-config-mount-outside-checkout

Conversation

@JasonWildMe

Copy link
Copy Markdown

Stops git from being able to delete a deployment's model registry.

The problem

docker-compose.prod.yml mounted the registry from inside the working tree — the only volume not sourced from .env:

- "${MODELS_DIR:?Set MODELS_DIR in .env}:/datasets:ro"      # from .env
- "${DATA_DB_DIR:?Set DATA_DB_DIR in .env}:/data/db:ro"     # from .env
- ../app/model_config.json:/app/app/model_config.json:ro    # <- from the CHECKOUT

The repo's app/model_config.json is a 5-model development stub. A real deployment serves a different, much larger registry. Because the mount points at a tracked path, the live file exists only as an uncommitted local edit — in no git history at all.

That makes routine git a delete button: git checkout, reset --hard, stash, or a pull touching that path silently replaces the live registry with the stub. The service then fails to start on the next restart, taking down every install that shares it.

This already happened

On 2026-07-16 a shared ml-service was reverted to main to back out an unrelated code change. Git restored its own 5-model stub over the 49-model production registry, and the service died on a stub entry (miewid-trout) whose weights don't exist on that host — after the revert had been made specifically to restore service.

The registry survived only because a copy happened to sit on someone's laptop.

The fix

- "${MODEL_CONFIG:?Set MODEL_CONFIG in .env}:/app/app/model_config.json:ro"

Now consistent with MODELS_DIR and DATA_DB_DIR. Verified with real docker compose: rejected when unset, resolves when set.

Git can't reach it checkout/reset/stash/pull all become harmless
Fails fast :? errors at compose time naming the variable, instead of silently mounting the stub
Stays private the registry never has to be committed to be deployed
Matches convention same shape as the other required volumes

Required, not defaulted — deliberately

Defaulting to the old path would keep the landmine armed for anyone who doesn't read the release note, and the failure it prevents is a multi-install outage.

Cost: every deployment must set MODEL_CONFIG before its next restart, or compose refuses to start. That's a loud error naming the exact variable — strictly better than a mysterious 5-model stub crash, which is what happens today.

Documentation

The old README actively taught the bad practice ("Edit app/model_config.json"), so this is fixed in three places:

  • docker/_envMODEL_CONFIG as required, with the reason
  • README.md — set it in .env; create the registry outside the checkout
  • app/model_config.json — a _note marking it a dev stub and warning against in-place server edits (main.py reads only "models", so the key is ignored)

Migration

sudo mkdir -p /data/ml-service
sudo cp <checkout>/app/model_config.json /data/ml-service/model_config.json
echo 'MODEL_CONFIG=/data/ml-service/model_config.json' >> docker/.env
# restart; confirm /health reports the expected models_loaded

What this does NOT do

It isn't a backup. It stops git eating the registry; it doesn't protect against losing the host. A file that five installs depend on, exists in no version control, and has one off-host copy is still a single point of failure — worth solving separately, mindful that the registry is not something to make public.

Related: per-install model configuration would also bound the blast radius, since main.py eager-loads every entry regardless of which install references it.

🤖 Generated with Claude Code

docker-compose.prod.yml mounted the registry from inside the working tree:

    - ../app/model_config.json:/app/app/model_config.json:ro

It was the only volume not sourced from .env, and it made `git` a delete button.
The repo's app/model_config.json is a 5-model development stub; a real deployment
serves a different, much larger registry. Keeping the live file at a tracked path
means it exists only as an uncommitted local edit, so `git checkout`, `reset
--hard`, `stash`, or a pull touching that path silently replaces it with the stub
-- and the service then fails to start on the next restart, taking down every
install that shares it.

This is not hypothetical. On 2026-07-16 a shared ml-service was reverted to main
to back out an unrelated code change; git restored its own 5-model stub over the
49-model production registry, and the service died on a stub entry whose weights
do not exist on that host. The revert was the right call for the original
problem; this line turned it into a second outage. The registry survived only
because a copy happened to exist on someone's laptop -- it is in no git history.

    - "${MODEL_CONFIG:?Set MODEL_CONFIG in .env}:/app/app/model_config.json:ro"

Now consistent with MODELS_DIR and DATA_DB_DIR: sourced from .env, and `:?` fails
at compose time with an actionable message rather than silently mounting the stub.
Verified with docker compose: rejected when unset, resolves when set.

Deliberately required rather than defaulted to the old path. A default would keep
the landmine armed for anyone who does not read the release note, and the failure
it prevents is a multi-install outage. The cost is that each deployment must set
MODEL_CONFIG before its next restart -- a compose error naming the variable, which
is far better than a mysterious stub crash.

Also keeps the registry private: it never has to be committed to be deployed.

Documented in three places, because the old README actively taught the bad
practice ("Edit app/model_config.json"):
- docker/_env: MODEL_CONFIG as required, with the reason
- README: set it in .env; create the registry outside the checkout
- app/model_config.json: a _note marking it a dev stub and warning against
  in-place server edits (main.py reads only "models", so the key is ignored)

Migration:
    sudo mkdir -p /data/ml-service
    sudo cp <checkout>/app/model_config.json /data/ml-service/model_config.json
    echo 'MODEL_CONFIG=/data/ml-service/model_config.json' >> docker/.env
    # restart; confirm /health reports the expected models_loaded

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant