refactor(metadata): finish wiring invoke formatter to pydantic union (#216)

* refactor(metadata): start to replace ad-hoc metadata parsing with structured parsing via pydantic

* feature(backend): continue implementation of pydantic parsing of invoke metadata

* refactor(metadata): add support for v5 metadata and v2 canbas

* fix(metadata): change opacity from int to float

* fix(metadata): change more opacity to float

* fix(metadata): handle version 3 postprocessed images

* refactor(metadata): finish wiring invoke formatter to pydantic union

The invoke metadata refactor left `invoke_formatter.py` pointing at
classes that no longer existed, so opening the drawer on an
InvokeAI-generated image would raise ImportError. This change completes
the wiring and locks in the behavior with regression tests.

- Add `InvokeMetadataView`, a version-agnostic facade over the
  `GenerationMetadata2/3/5` discriminated union, so the formatter
  doesn't sprinkle `isinstance` checks across extraction logic.
- Rewrite `invoke_formatter.py` as a thin HTML renderer on top of the
  view, preserving the existing drawer table layout.
- Move `_normalize_ref_images` out of `GenerationMetadataAdapter.parse`
  and into `GenerationMetadata5` as a `@model_validator(mode="before")`,
  where it belongs.
- Re-enable `metadata_modules/__init__.py` re-exports that had been
  commented out during the refactor.
- Delete the now-unused `invoke/invoke_metadata_abc.py`.
- Drawer HTML tweaks: suppress empty negative-prompt row; keep positive
  prompt row but drop copy icon when empty; per-column suppression in
  tuple tables; fall back to `1.0` when a surviving weight column has
  gaps; surface Flux Redux `imageInfluence` (e.g. "Medium") in the
  weight column when no numeric weight is present.
- Add 33 regression tests in `tests/backend/test_invoke_metadata.py`
  covering view-level extraction and end-to-end HTML rendering for
  v2/v3/v5 (both `ref_images` and `canvas_v2_metadata` paths).
- Run ruff across `invoke/` to modernize `typing.Optional`/`Union`/
  `List`/`Dict` to PEP 604 / PEP 585 forms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: lstein

CLAUDE.md

@@ -0,0 +1,80 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project
+
+PhotoMapAI is a local-first image browser for large photo collections. It uses CLIP embeddings to power semantic text/image search and builds a UMAP "semantic map" that clusters images by content. The backend is FastAPI; the frontend is vanilla ES6 modules (no framework) using Swiper.js and Plotly.js. All processing is local — nothing is sent to external services.
+
+## Common Commands
+
+```bash
+# Install for development (Python 3.10–3.13)
+pip install -e .[testing,development]
+npm install
+
+# Run the server (entry point defined in pyproject.toml)
+start_photomap               # http://localhost:8050
+
+# Tests
+make test                    # runs npm test + pytest
+pytest tests                 # backend only
+pytest tests/backend/test_search.py::test_text_search    # single test
+npm test                     # frontend Jest only
+NODE_OPTIONS='--experimental-vm-modules' jest tests/frontend/search.test.js  # single JS test
+
+# Linting / formatting (CI enforces both)
+make lint                    # runs backend-lint + frontend-lint
+ruff check photomap tests --fix
+npm run lint:fix
+npm run format               # prettier write
+npm run format:check         # CI check
+
+# Docs
+make docs                    # mkdocs serve on :8000
+```
+
+Ruff is configured for line-length 120, target py310, rules E/W/F/I/UP/B (see pyproject.toml). Jest runs in jsdom with experimental ESM (the project is `"type": "module"`).
+
+## Architecture
+
+### Backend layout (`photomap/backend/`)
+
+- `photomap_server.py` — FastAPI app entry point. Wires up routers, mounts `/static` and Jinja2 templates, and defines the top-level `/` route. `start_photomap` from `pyproject.toml` runs `main()` here.
+- `routers/` — one router per API surface: `album`, `search`, `umap`, `index`, `curation`, `filetree`, `upgrade`. Routers are included in `photomap_server.py`; `curation_router` is mounted with an explicit `/api/curation` prefix while the others set their own prefixes.
+- `config.py` — YAML-backed album config. Access via the `get_config_manager()` singleton (lru_cached). `Album` is a Pydantic model that expands `~` in image paths. Config lives in a platformdirs user config directory.
+- `embeddings.py` — CLIP embedding generation and persistence (`.npz`).
+- `imagetool.py` — shared CLI entry point for `index_images`, `update_images`, `search_images`, `search_text`, `find_duplicate_images` (all registered as scripts in `pyproject.toml`).
+- `metadata_extraction.py` / `metadata_formatting.py` — pulls EXIF + generator metadata (InvokeAI) out of images and formats for the UI.
+
+### Metadata subsystem (`photomap/backend/metadata_modules/`)
+
+This is the area under active refactor (current branch: `lstein/feature/refactor-invoke-metadata`). InvokeAI writes several incompatible metadata schemas into PNG tEXt chunks; the parser must auto-detect and upgrade.
+
+- `invokemetadata.py` defines `GenerationMetadata` as a Pydantic `Annotated[Union[…], Field(discriminator="metadata_version")]` over `GenerationMetadata2`, `GenerationMetadata3`, and `GenerationMetadata5`. `GenerationMetadataAdapter.parse()` inspects fields like `canvas_v2_metadata`, `app_version`, and `model_weights` to inject the correct `metadata_version` when the source JSON predates the discriminator.
+- `invoke/` holds the per-version schemas: `invoke2metadata.py`, `invoke3metadata.py`, `invoke5metadata.py`, plus `canvas2metadata.py` and `common_metadata_elements.py` for shared types. `invoke_metadata_view.py` is the version-agnostic facade consumed by `invoke_formatter.py`. When adding support for a new InvokeAI version, add a new `invokeNmetadata.py`, extend the Union in `invokemetadata.py`, teach `parse()` how to recognize legacy payloads that lack a `metadata_version` field, and extend `InvokeMetadataView`'s `isinstance` dispatch.
+- `invoke_formatter.py` / `exif_formatter.py` render parsed metadata for the drawer UI; `slide_summary.py` produces the compact slideshow caption.
+- `invoke-DELETE/` is a holdover from the refactor — leave it alone unless cleaning up.
+
+### Frontend layout (`photomap/frontend/`)
+
+- `static/javascript/` — one ES6 module per feature. No build step; modules are served directly and imported from `main.js` / `index.js`.
+- `state.js` is the centralized application state. Prefer extending it over adding new globals.
+- `events.js` owns global keyboard shortcuts; register new ones there rather than scattering listeners.
+- `localStorage` is used for persisted user preferences, `sessionStorage` for per-navigation state.
+- `templates/` — Jinja2 templates rendered by FastAPI.
+
+### Tests
+
+- `tests/backend/` — pytest. `conftest.py` + `fixtures.py` set up shared fixtures (test images in `tests/backend/test_images/`). Use the FastAPI `TestClient` for router tests; see `test_search.py`, `test_albums.py`, `test_curation.py` as templates.
+- `tests/frontend/` — Jest with jsdom. `setup.js` provides DOM fixtures. See `tests/frontend/README.md` for setup notes.
+
+## Conventions to follow
+
+From `.github/copilot-instructions.md` — the parts that actually affect how you write code here:
+
+- **Python:** type hints on public functions, `pathlib.Path` (not `os.path`) for file operations, f-strings, imports ordered stdlib → third-party → local (`photomap` is first-party to isort). Code must pass `ruff check photomap tests`.
+- **Pinned quirk:** `setuptools<67` is intentional — avoids a deprecation warning from the CLIP dependency. Don't "fix" it.
+- **New API endpoints:** add/extend a router under `photomap/backend/routers/`, use Pydantic models for request/response, include the router in `photomap_server.py`, add a `tests/backend/test_<name>.py`.
+- **New frontend features:** create a module in `static/javascript/`, wire shared state through `state.js`, register shortcuts in `events.js`, add a Jest test.
+- **JavaScript:** ES6 modules only, `const`/`let`, must pass `npm run lint` and `npm run format:check`.

photomap/backend/metadata_modules/__init__.py

@@ -1,4 +1,3 @@
-
 from .exif_formatter import format_exif_metadata
 from .invoke_formatter import format_invoke_metadata
 from .slide_summary import SlideSummary

photomap/backend/metadata_modules/invoke-DELETE/__init__.py

@@ -0,0 +1,10 @@
+# from .legacy import InvokeLegacyMetadata
+# from .v3 import Invoke3Metadata
+# from .v5 import Invoke5Metadata
+
+# # reexport the main classes
+# __all__ = [
+#     "InvokeLegacyMetadata",
+#     "Invoke3Metadata",
+#     "Invoke5Metadata",
+# ]

…ta_modules/invoke/invoke_metadata_abc.py …les/invoke-DELETE/invoke_metadata_abc.pyphotomap/backend/metadata_modules/invoke/invoke_metadata_abc.py renamed to photomap/backend/metadata_modules/invoke-DELETE/invoke_metadata_abc.py photomap/backend/metadata_modules/invoke/invoke_metadata_abc.py renamed to photomap/backend/metadata_modules/invoke-DELETE/invoke_metadata_abc.py

@@ -0,0 +1,125 @@
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field, model_serializer, model_validator
+
+from photomap.backend.metadata_modules.invoke.common_metadata_elements import (
+    ControlAdapter,
+    Fill,
+    Object,
+    Position,
+    ReferenceImage,
+    RegionalGuidance,
+    tag_reference_images,
+)
+
+
+class Clip(BaseModel):
+    height: float
+    width: float
+    x: float
+    y: float
+
+
+class Inpaintmask(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+    fill: Fill
+    id: str
+    is_enabled: bool = Field(alias="isEnabled")
+    is_locked: bool = Field(alias="isLocked")
+    name: Any | None
+    objects: list[Object]
+    opacity: float
+    position: Position
+    type: str
+
+
+class Rasterlayer(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+    id: str
+    is_enabled: bool = Field(alias="isEnabled")
+    is_locked: bool = Field(alias="isLocked")
+    name: Any | None
+    objects: list[Object]
+    opacity: float
+    position: Position
+    type: str
+
+
+class ControlLayer(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+    control_adapter: ControlAdapter = Field(alias="controlAdapter")
+    id: str
+    is_enabled: bool = Field(alias="isEnabled")
+    is_locked: bool = Field(alias="isLocked")
+    name: Any | None
+    objects: list[Object]
+    opacity: float
+    position: Position
+    type: str
+    with_transparency_effect: bool = Field(alias="withTransparencyEffect")
+
+
+class CanvasV2Metadata(BaseModel):
+    model_config = ConfigDict(populate_by_name=True)
+    raster_layers: list[Rasterlayer] | None = Field(None, alias="rasterLayers")
+    control_layers: list[ControlLayer] | None = Field(None, alias="controlLayers")
+    inpaint_masks: list[Inpaintmask] | None = Field(None, alias="inpaintMasks")
+    reference_images: list[ReferenceImage] | None = Field(
+        None, alias="referenceImages"
+    )
+    regional_guidance: list[RegionalGuidance] | None = Field(
+        None, alias="regionalGuidance"
+    )
+
+    @model_validator(mode="before")
+    @classmethod
+    def _preprocess_canvas_metadata(
+        cls, canvas_metadata: dict[str, Any]
+    ) -> dict[str, Any]:
+        """Preprocess canvas metadata to add type discriminators to image objects."""
+
+        def process_image_in_dict(obj: dict[str, Any], key: str = "image") -> None:
+            """Add type discriminator to an image object if it exists."""
+            if key in obj and obj[key]:
+                tag_reference_images(obj[key])
+
+        def process_objects(objects: list[dict[str, Any]]) -> None:
+            """Process a list of objects that may contain images."""
+            for obj in objects:
+                process_image_in_dict(obj)
+
+        def process_reference_images(ref_images: list[dict[str, Any]]) -> None:
+            """Process reference images with ipAdapter."""
+            for ref_image in ref_images:
+                if "ipAdapter" in ref_image and ref_image["ipAdapter"]:
+                    process_image_in_dict(ref_image["ipAdapter"])
+
+        # Process layers with objects (rasterLayers, inpaintMasks, controlLayers)
+        for layer_key in ["rasterLayers", "inpaintMasks", "controlLayers"]:
+            if layer_key in canvas_metadata and canvas_metadata[layer_key]:
+                for layer in canvas_metadata[layer_key]:
+                    if "objects" in layer and layer["objects"]:
+                        process_objects(layer["objects"])
+
+        # Process top-level referenceImages
+        if "referenceImages" in canvas_metadata and canvas_metadata["referenceImages"]:
+            process_reference_images(canvas_metadata["referenceImages"])
+
+        # Process regionalGuidance with objects and referenceImages
+        if (
+            "regionalGuidance" in canvas_metadata
+            and canvas_metadata["regionalGuidance"]
+        ):
+            for region in canvas_metadata["regionalGuidance"]:
+                if "objects" in region and region["objects"]:
+                    process_objects(region["objects"])
+                if "referenceImages" in region and region["referenceImages"]:
+                    process_reference_images(region["referenceImages"])
+
+        return canvas_metadata
+
+    @model_serializer(mode="wrap")
+    def serialize_model(self, serializer, info):
+        """Exclude None values when serializing."""
+        data = serializer(self)
+        return {k: v for k, v in data.items() if v is not None}