Spec: robust CSV metadata ingestion - cb tagging fix + CSV-to-DB pipeline + CLAP for genre-less tracks

[brchn6]

Problem Statement

The DJ metadata pipeline loses the annotation data that already exists. Exportify CSVs carry real Spotify metadata (genre, key, energy, danceability, popularity, label, URI) for every gig track, but today that data does not reliably reach Mixxx:

1. cb silently drops the annotation at fetch time: fetch_csv reads the CSV fields correctly and calls _tag_file(genre=..., key=..., tempo=...), but mutagen is missing in cb's runtime environment (miniconda). _tag_file catches the ImportError and prints "mutagen not available - skipping tagging" - while the fetch log still shows "✓ Fetched". Result: files carry yt-dlp's YouTube-category genres ("Music", "Entertainment"), and that garbage flows into MultiDJ and Mixxx.
2. MultiDJ import clobbers sourced genres: directory.py import overwrites genre from file ID3 tags unconditionally, even when the DB track already has an authoritative genre_source. A CSV→DB ingest that set real genres was wiped on the next sync (fixed in commit 129e9f9, needs a regression test).
3. The gig workflow is one-off manual: the CSV→DB ingest (dj-csv-ingest.py) was written as a one-time repair for Erin_gig, not wired into the standard gig pipeline. Future gigs would repeat the bug.
4. 54 Erin_gig tracks have no genre at all: their Exportify rows have empty Genres (Spotify has no genre), and MusicBrainz returns nothing. They show "Music" in Mixxx. The CLAP classifier in enrich_genre could classify them from audio - it already has the model and 5,471 embeddings - but those 54 tracks have no embeddings yet.

Solution

Make the CSV metadata flow end-to-end and automatically:

1. Fix cb's tagging so future fetches write real genres/keys/tempos into file tags (install mutagen into miniconda, cb's actual runtime).
2. Harden MultiDJ import so a track with genre_source set never has its genre overwritten by file-tag noise (done in 129e9f9; add regression tests).
3. Wire dj-csv-ingest.py into the standard gig pipeline (dj-fetch.sh), so every Exportify CSV's metadata lands in the MultiDJ DB automatically - genre, Camelot key, energy, label, release year, and full track_tags (URI, danceability, popularity, explicit, tempo, loudness...).
4. Give the 54 genre-less tracks real genres via CLAP: embed those tracks, then run enrich_genre (file→Discogs→MusicBrainz→CLAP layered pipeline, all in multidj). Optionally enrich the whole library's genre-less tracks the same way.

This turns the CSV from a manual Exportify artifact into one input format of a robust ingestion layer - the first step toward "engulfing Exportify" (Spotify API → same DB writes, no web toggling).

User Stories

1. As a DJ, I want cb to write the real Spotify genre/key/tempo into file tags when fetching from an Exportify CSV, so that the annotation I already have survives the download.
2. As a DJ, I want cb to never silently skip tagging, so that I can trust a "✓ Fetched" line to mean the metadata was written.
3. As a DJ, I want MultiDJ import to preserve a sourced genre, so that a re-import never clobbers real metadata with YouTube-category noise.
4. As a DJ, I want every gig fetch to automatically ingest CSV metadata into the MultiDJ DB, so that genres/keys/labels are correct in Mixxx without manual repair scripts.
5. As a DJ, I want the CSV ingest to match tracks even when dedupe kept a survivor library copy, so that the gig playlist (which points at survivors) gets the metadata.
6. As a DJ, I want the 54 tracks with no Spotify genre to get a genre from audio analysis (CLAP), so that every track in my gig playlist has a usable genre.
7. As a DJ, I want to see provenance (genre_source, genre_confidence) for every genre, so that I know whether a genre came from Spotify, Discogs, MusicBrainz, CLAP, or a manual tag.
8. As a DJ, I want the ingest to be idempotent and re-runnable, so that re-running a gig fetch does not duplicate or corrupt metadata.
9. As a DJ, I want the pipeline to respect the Mixxx session lock, so that DB writes never collide with a running Mixxx session.
10. As a DJ, I want the CSV ingest to keep all the extra Exportify fields (URI, danceability, popularity, explicit, tempo, loudness) in track_tags, so that future features (smart playlists, energy sorting, set building) can use them.
11. As a DJ, I want the 3 known failed downloads (yt-dlp 403/age) to be reported clearly, so that I know which tracks need manual attention.
12. As a DJ, I want a dry-run mode on the CSV ingest, so that I can preview matches and genres before writing anything.
13. As a DJ, I want the genre-less tracks that CLAP cannot classify confidently to stay untouched, so that no wrong genres are written.
14. As a DJ, I want the whole genre enrichment (Discogs/MusicBrainz/CLAP) runnable against all genre-less library tracks, so that the library keeps getting more robust over time.
15. As a developer, I want regression tests for the import-preserves-sourced-genre behavior, so that the fix is not silently undone.
16. As a developer, I want unit tests for the CSV ingest matching logic, so that filename reconstruction and survivor fallback are covered.
17. As a developer, I want the enrich_genre path tested end-to-end (Discogs hit, MusicBrainz hit, CLAP fallback, no-API fallback), so that the layered pipeline is trustworthy.

Implementation Decisions

• cb fix (separate repo: Clouduccaneer): install mutagen into the miniconda env that cb actually resolves to (/home/barc/miniconda3). Do NOT install into the multidj venv - cb does not run from there. Optionally make _tag_file fail loudly (log line + non-zero marker) instead of a silent skip when mutagen is unavailable, so the "✓ Fetched" lie cannot recur.
• MultiDJ import hardening: directory.py already keeps genre when genre_source is set (commit 129e9f9). Extend the same principle if any other file-tag-derived field gains a source later. The DB is the source of truth; file tags are one input.
• CSV ingest in the standard pipeline: dj-csv-ingest.py (already built for Erin_gig) becomes a phase in dj-fetch.sh after cb fetch-csv and before/around dj-sync.sh. Input: gig name + CSV dir; output: MultiDJ DB rows + track_tags. Matching strategy (already proven): reconstruct cb's exact filename (album - artist - title, safe-named) against the Mixxx playlist paths (which point at dedupe survivors), with a component-based fallback for pre-convention names.
• CLAP for genre-less tracks: run multidj analyze embed (CLAP) for tracks lacking embeddings, then multidj pipeline --phase enrich (enrich_genre uses file→Discogs→MusicBrainz→CLAP). CLAP minimum confidence threshold already exists (0.25). Discogs requires a user token in the multidj config (discogs.token) - the user adds it themselves, never through chat; discogs_client must be installed in the multidj venv.
• Schema: no new columns needed - genre_source, genre_confidence, release_year, label exist; track_tags holds the rest. Key stored as Camelot (mapped from Spotify key+mode via the existing constant map).
• Genre shape: store the first (primary) CSV genre in genre; keep the full comma-joined list in track_tags.genres_full so Mixxx genre crates (which group on exact genre) work while the full list stays available.

Testing Decisions

• What makes a good test: assert external behavior through the public functions - a genre that had a source survives import; a CSV row maps to the right track; enrich_genre returns the right source/confidence for each layer. Never re-implement the logic inline (repo anti-pattern #4).
• Import hardening: extend tests/test_import_directory.py (existing seam: make_multidj_db fixture + mutagen-tagged temp files). New test: track with genre_source='spotify-exportify' and genre "mizrahi" is imported over a file tagged "Music" - genre must stay "mizrahi", source unchanged.
• CSV ingest: new tests/test_csv_ingest.py using the existing multidj_db fixture: (a) exact filename reconstruction matches a playlist-path track; (b) component fallback matches a pre-convention name; (c) dry-run writes nothing; (d) apply writes genre/source/confidence and track_tags; (e) idempotent on re-run; (f) lock-active blocks writes. The matching functions are pure (filename reconstruction, normalization, fallback scoring) - test those directly and the DB writes through the existing fixture.
• enrich_genre: prior art is tests/test_enrich_genre.py (mock-free, sets genres directly in the fixture DB). Add: MusicBrainz-hit path, CLAP-fallback path with a stub embedding vector, and "no API hit and no specific genre → stays untouched" path. Embedding-dependent tests use a small hand-made vector blob in the fixture DB.
• Pipeline wiring: a smoke test that dj-fetch.sh --dry parses and reports without side effects (shell-level, mirrors how dj-playlist verification is done).

Out of Scope

• Engulfing Exportify (replacing the manual CSV export with direct Spotify API calls) - the ingest layer built here is the foundation, but the Spotify-API fetch path is a separate feature.
• Re-tagging the audio files' ID3 tags retroactively for the existing 251 Erin_gig tracks (their metadata now lives correctly in the DB; file tags are not the source of truth).
• Recovering the 3 failed Erin_gig downloads (yt-dlp 403/age) - requires manual source re-fetch.
• Color tagging in Mixxx (separate feature mentioned by the user, out of scope here).
• BPM/key automation for gig files (deliberately manual - user detects in Mixxx GUI).

Further Notes

• Executed background: dj-csv-ingest.py repaired Erin_gig on 2026-08-13 - 280/283 CSV rows matched, 201/255 playlist tracks got real genres (56 distinct), 54 have empty Genres in Spotify's data, 3 unmatched are the known failed downloads. This spec hardens and generalizes that work.
• The db backups are taken before every write path (dj-sync.sh and the ingest both backup); never delete anything; respect the Mixxx session lock (dj.lock, 6h stale).
• Discogs token goes in ~/.multidj/config.toml by the user - zero-trust rule: credentials never pass through chat or agent files.