Incremental UMAP for growing datasets: --transform-from, growing --align, --register-to

[enjalot]

Closes #142.

A daily-growing dataset can now keep a visually stable published map:

• ls-umap <ds> <emb> --transform-from <umap-id> — loads the source umap's saved (--save) reducer pickle (following the reducer_id chain when the source was itself a transform output), projects only the rows appended since the source, and maps them into the source's [-1,1] frame via its stored min/max — or via its registration transform if the source was registered, so the daily cadence keeps working after a registered refit. Old rows are copied verbatim (byte-identical) so published positions never move; out-of-frame new points are left in place and counted. Pickles are never copied.
• Growing-window --align — identity relations replaced by shared-prefix relations {j: j for j in range(min(len_i, len_i+1))}; identical to old behavior for equal lengths, supports append-only windows.
• --register-to <umap-id> — registers a plain or aligned fit onto an existing umap with a least-squares 2D similarity transform (Umeyama; rotation + uniform scale + translation, reflections allowed) fit on the shared row prefix. Output is already in the target's frame (no renormalization); meta records registered_to + the transform.
• docs/umap.md — the daily/periodic recipe, plus the --seed→single-threaded-fit caveat and reducer-pickle size/pruning notes from the issue.

Both new flags are forwarded by the server job route. Pure numpy pieces live in latentscope/scripts/registration.py.

Recipe:

ls-umap mydata embedding-001 25 0.1 --save                       # once: fit + save reducer
ls-umap mydata embedding-001 --transform-from umap-001           # daily: project new rows
ls-umap mydata embedding-002 25 0.1 --align embedding-001 --register-to umap-001 --save  # periodic refit

Test plan

• uv run pytest tests/ -q — 233 passed (14 new: umeyama recovery incl. reflection/degenerate cases, prefix relations for unequal windows, normalization, and the full transform flow via a fake pickled reducer — success, chain resolution, registered source, missing pkl, mismatched embedding, no-new-rows)
• Verified end-to-end with real UMAP/AlignedUMAP fits on a 120-row dataset: daily transform kept 100 old rows byte-stable and appended 20; growing align (100→120) + register wrote both slices anchored to the published frame
• Behavior with none of the new flags is unchanged

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ea806be31

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Persist a reducer for the aligned refit

In the periodic refit workflow this command accepts --save, but the align branch in latentscope/scripts/umapper.py returns before the only if save: pickle.dump(...) path, so it writes the registered umap-NNN.json/parquet without a matching .pkl or reducer_id. If the user then follows the documented daily cadence and runs --transform-from on that refit, _resolve_reducer_id finds no saved reducer and the incremental workflow fails; either persist a usable reducer/reducer reference for aligned outputs or avoid documenting --save here.

Useful? React with 👍 / 👎.

[enjalot]

Fixed in 6e2ec89 — --align --save now pickles a reducer per aligned output. Two subtleties the fix handles: (1) AlignedUMAP's mappers_[i].embedding_ lives in the mapper's own frame, not the aligned frame (empirically unrelated — mean deviation ~78% of layout span), so the slice's aligned coordinates are swapped into embedding_ before pickling; UMAP.transform embeds new points relative to embedding_, which was verified to re-place training rows within ~9% of span of their aligned positions (normal transform stochasticity). (2) The meta's min/max or registration transform then maps transformed points into the published frame, exactly as --transform-from expects. Test added: align+save writes a loadable pkl per slice whose embedding_ is the aligned frame, and --transform-from on an aligned output resolves its reducer.