Unified collection import/export format; LMX collections distributed via Hugging Face

[jeremyfowers]

What

Makes omni collections (recipe: collection.omni) behave like virtual models: they import/export through the exact same pipeline as regular models, using one JSON format shared by the server, GUI, CLI, and Hugging Face.

The format

A collection file is the regular-model export format plus components (ordered names) and models (each component's full definition, normalized by the same per-model transform):

{
    "model_name": "user.LMX-Omni-5.5B-Lite",
    "recipe": "collection.omni",
    "checkpoints": { "main": "lemonade-sdk/LMX-Omni-5.5B-Lite" },
    "components": ["Qwen3.5-4B-MTP-GGUF", "SD-Turbo", "Whisper-Tiny", "kokoro-v1"],
    "models": [ { "model_name": "Qwen3.5-4B-MTP-GGUF", "recipe": "llamacpp", "checkpoints": { "...": "..." }, "labels": ["..."], "size": 3.66 }, "..." ],
    "labels": [],
    "recipe_options": {},
    "size": 9.3
}

The same file works verbatim as: a POST /v1/pull body, a lemonade import file, and an HF-hosted manifest (<CollectionName>.json).

Example Model Files

https://huggingface.co/lemonade-sdk/LMX-Omni-52B-Halo
https://huggingface.co/lemonade-sdk/LMX-Omni-5.5B-Lite

Changes

• Server read side: /v1/models and /v1/models/{id} embed an ordered models[] array for collections (each component's full model object, parallel to components).
• Server pull side: /v1/pull accepts the exported file. Unknown components are registered from the embedded models[] definitions; known names keep the local definition (local-wins, drift logged as a warning). The existing component fan-out, cycle guard, and progress streaming are unchanged.
• HF distribution: the built-in LMX-Omni-52B-Halo / LMX-Omni-5.5B-Lite entries are slim stubs (checkpoint = HF repo); pulling fetches the manifest from the repo, discovered by content (the repo's *.json with recipe: collection.omni) rather than a hardcoded filename. Manifests cache in the HF cache like any model file, so collections survive restarts and work offline after first pull.
• CLI: lemonade export/import work for collections via the shared validate_and_transform_model_json transform (id→model_name, kKnownKeys allow-list, checkpoint dedup, per-element normalization). Regular-model export output is byte-identical to before.
• GUI: collection and regular-model export both fetch /models/{id} and save the normalized file (shared helper mirroring the CLI transform); collection import is a single POST /pull; export filenames drop the user. prefix; the legacy {version, collections, models} bundle format is removed.
• Exports strip user-specific state: suggested, created, downloaded never appear in files (top level or inside models[]); the server regenerates them on import.
• Docs: format documented on /v1/pull (lemonade.md, incl. a "Collection Files: Export, Import, and Hugging Face" section) and the model object (openai.md).

Breaking changes

• Previously exported GUI collection bundles ({version, exportedAt, collections, models}) no longer import (hard cutover; the GUI was the only producer/consumer).

Verification

• server_endpoints.py: 50/50 OK · server_cli2.py: 58/58 OK (1 skipped) · server_omni.py --wrapped-server llamacpp --backend vulkan: 7/7 OK (pre-pulls LMX-Omni-5.5B-Lite through the HF manifest path, then drives the server-side orchestration loop)
• App regression suite: 31/31; collection unit tests rewritten for the new transform
• Round-trips verified live: export ≡ uploaded HF manifest (JSON-equal); fresh-cache pull of both LMX collections; import of a collection file with an unknown component (registered as user.* from the embedded def, downloaded, survives restart); verbatim POST /pull of an exported file
• Builds: C++ (Ninja Release) and web-app clean

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0f929003f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Persist refreshed collection components

When an already-registered HF-backed user collection is pulled again after its manifest changes, this branch fetches and downloads the refreshed component list but never writes components back to user_models.json. The cache invalidation below then rebuilds user collections from the persisted stale components (only built-in empty-component collections call populate_collection_components_from_cache_locked), so /models, load/routing, and downloaded status continue to reflect the old collection even though the new manifest was fetched.

Useful? React with 👍 / 👎.

[fl0rianr]

Thanks, this is a good direction overall: sharing one import/export shape across the server, CLI, GUI, and HF manifests should simplify the collection story a lot.

Before merge let's fix two correctness issues:

1. I agree with the existing Codex inline comment on src/cpp/server/model_manager.cpp:3139: when an already-registered HF-backed user collection is refreshed from a changed manifest, the refreshed component list is used for the current download but is not persisted back into user_models.json. After cache invalidation/rebuild, the registered collection can still expose/load the stale components. Please persist the canonical refreshed components for user.* collections after manifest resolution, and add a regression test for manifest A/B -> A/B/C refresh.

2. Inline collection imports should fail closed when components and models do not match. The format is documented as ordered/parallel; silently dropping missing or invalid component definitions can create a different collection than the exported file describes.

[fl0rianr]

This should probably be a hard import failure for inline collection files, not a warning/skip. validate_collection_request() only checks that models is an array; after that, any unknown component without a matching valid definition is skipped here. If at least one component remains, the collection can be persisted with a shortened component list, which is not the same collection the exported file described.

Can we require models to match components exactly for inline imports (same names, no missing definitions, no invalid entries), and abort/rollback instead of silently dropping components?

[fl0rianr]

The implementation here requires the manifest to be named exactly .json. That matches the new user docs, but the PR description says HF manifests are discovered "by content" as the repo's *.json with recipe: collection.omni.