MISSION: Library-wide CLAP genre labeling (2,533 unlabeled tracks)

[brchn6]

Mission: Library-wide CLAP genre labeling (all unlabeled tracks)

Created: 2026-08-15 (after the 54-track PoC, README.md in this dir)
Owner: next session
Status: ready to execute
Scope: 2,533 active tracks with no usable genre (2,290 junk "Music" + 243 empty),
verified by query - total audio 19.5 GB, 0 missing files.

---

Goal

Give every unlabeled track a coarse but real CLAP genre (genre_source='clap',
genre_confidence = raw top-1 cosine), merged into the MultiDJ DB and synced to
Mixxx - so genre crates work for the whole library, not just Erin_gig.

Phase 0 - Curated vocab (code change, local sandbox, then ship)

The 61-label DJ_GENRE_LABELS causes wrong-adjacent picks (Pitbull->Funk,
Bad Bunny->Afrobeats, 9094 disco->Classical). CLAP discriminates ~20 broad
acoustic families; cultural subgenres (mizrahi, dembow, Miami bass) are a
manual-refinement layer, not a CLAP job.

Proposed curated list (20)

#	Label	Absorbs (removed labels)
1	Pop	-
2	Dance Pop	-
3	Hip-Hop	Trap, Pop Rap, Latin Trap
4	R&B / Soul	R&B, Soul
5	Reggaeton / Latin	Reggaeton, Latin Pop, Latin Trap, Bachata, Salsa
6	Rock / Indie	Rock, Indie Pop, Synthpop, Alternative
7	House	Deep/Tech/Melodic/Organic House, Garage House
8	Techno	Minimal/Acid/Industrial/Melodic Techno
9	Trance / Progressive	Progressive Trance, Trance
10	Drum & Bass	-
11	Dubstep / Bass	Dubstep
12	Afro House / Amapiano	Afro House, Amapiano
13	Afrobeats	-
14	Disco / Nu-Disco	Disco, Nu-Disco
15	Funk	-
16	Reggae / Dancehall	Reggae, Dancehall
17	EDM / Festival	EDM, Electro House, Big Room House
18	Eurodance	-
19	Downtempo / Ambient	Downtempo, Ambient
20	Latin House	- (real crate in the library)

Explicitly dropped (why)

• Classical - stole the disco track 9094; no classical in this library.
• Mizrahi / Israeli Pop - CLAP has no acoustic category for them; mizrahi
  tracks will land on Pop/Hip-Hop. Accept + refine manually (see Risks).
• Baile Funk / Samba / Forro / Brazilian Funk - CLAP over-uses these for
  latin percussion; one "Reggaeton / Latin" anchor instead.
• Party - negative separation in the PoC (-0.015); too vague.
• Ballroom, Country, Jazz, Ambient(lone) - no acoustic pull in this library.
• UK Garage - rare here; covered by House.

Regression gate (BEFORE the mission)

Re-run the 54-track PoC with the curated vocab and REQUIRE:

• 9094 (הדורבנים שוב הדיסקו) -> Disco / Nu-Disco (was Classical) - the
  canonical wrong-label test
• Pitbull tracks (8884/8960/9031) -> Reggaeton / Latin or Dance Pop,
  never Funk
• Bad Bunny (9041) -> Reggaeton / Latin, never Afrobeats
• Assigned count stays >= 50/54

Phase 1 - Build the id list

MUSIC="/home/barc/Weizmann Institute Dropbox/Bar Cohen/Music"
/home/barc/dev/multidj/.venv/bin/python - "$MUSIC" <<'PY'
import sqlite3, sys
junk = ("music","people & blogs","entertainment","comedy","sports","travel & events",
        "film & animation","gaming","education","howto & style","news & politics",
        "science & technology","autos & vehicles","pets & animals","nonprofits & activism")
c = sqlite3.connect(f"{sys.argv[1]}/.multidj/library.sqlite")
ids = [r[0] for r in c.execute(f"""SELECT id FROM tracks WHERE deleted=0 AND (
  genre IS NULL OR trim(genre)='' OR lower(trim(genre)) IN ({','.join('?'*len(junk))}))""", junk)]
open("ids_all.txt","w").write("\n".join(map(str, ids)) + "\n")
print(len(ids), "ids")
PY

Phase 2 - Batch + ship (6 batches, ~3.3 GB each)

split -n l/6 ids_all.txt batch/ids_ - then per batch the PoC flow:

1. run_poc.sh prep generalized: DB copy + path rewrite (REMOTE_PREFIX slice
   bug already fixed) + rsync batch mp3s (use --files-from like the PoC).
2. Ship repo (with curated vocab) + job/ scripts - re-ship after any edit.

Phase 3 - WEXAC setup (once)

• uv sync --extra embeddings then pin the torch family together:

  uv pip install --python .venv/bin/python \
    'torch==2.7.1+cu126' 'torchvision==0.22.1+cu126' 'torchaudio==2.7.1+cu126' \
    --index-url https://download.pytorch.org/whl/cu126
  uv pip install --python .venv/bin/python laion-clap

  (Order matters: laion-clap/torchvision after the pin will NOT re-break it if
  re-pinned after; verify import torch, torchvision, torchaudio, laion_clap.)
• Verify once on a GPU node: torch.cuda.is_available() == True (probe job).

Phase 4 - LSF GPU jobs (per batch)

Same poc_job.sh + poc_classify.py (per-file encoding - never batch the
audio call; raw-cosine confidence, --min-conf 0.30). Queue short-gpu,
-gpu 'num=1:j_exclusive=no:gmem=8G', -R 'rusage[mem=16000]'.
Runtime estimate: ~0.3 s/track on H200 -> ~15-20 min per 420-track batch.

Run batches SEQUENTIALLY (each writes to the same DB copy - the merge comes
back per batch and the real DB must not double-apply).

Phase 5 - Fetch + merge (per batch)

merge_results.py <fetched DB copy> - backs up the real DB before applying
(first batch only needs the backup; subsequent merges reuse the pattern but
keep one backup per merge is safest). Verify counts after each merge:
genre_source='clap' count == batch size minus unassigned.

Phase 6 - Sync + verify + dashboard v2

• ~/.pi/agent/skills/dj-sync/scripts/dj-sync.sh (lock-aware)
• Mixxx check: count tracks with real genres; junk-genre count should drop to ~0
• Dashboard v2: same builder (build_dashboard.py), now with up to 2,533 dots
  • UMAP on all labeled tracks (the big picture: do crates now show clusters?)
  • separation stats per genre; before/after table with inline audio samples
• Report: assigned/unassigned per batch, genre distribution, worst-confidence
  tracks (the honest-unknown list)

Acceptance criteria

• 2,290 junk-genre tracks + 243 empty -> real genres, genre_source='clap'
• Unassigned (conf < 0.30) stay empty or NULL - honest, never faked
• The 20-label regression gate passes (9094 disco, Pitbull latin, Bad Bunny latin)
• One environment (WEXAC) used for ALL batches - no mixing with local runs
  (borderline labels are machine-dependent; mixing breaks comparability)
• Mixxx Erin_gig + library crates show real genres

Risks / notes

• Mizrahi + Israeli pop cannot be labeled by CLAP - they will land on
  Pop/Hip-Hop. The manual refinement path (user bulk-edits in Mixxx/DB) is the
  fix; consider a follow-up ticket: mizrahi detection via lyrics/hebrew artist
  heuristics (multidj enrich_language already flags Hebrew tracks).
• 19.5 GB transfer - 6 batches of ~3.3 GB; LAN + WEXAC ingress should do
  ~10-20 min per batch. If slower than expected, raise batch count.
• Borderline flips - same fixed env keeps them consistent; re-runs with the
  same code+env are reproducible (verified per-file determinism).
• ~8-15% of labels will be coarse-wrong (PoC observed). The confidence
  column + dashboard make them findable; the audio-sample players let the user
  audit quickly.

Cleanup

After the mission: ssh login4 "rm -rf ~/dj-clap-poc" (audio copies + venv +
checkpoint cache) - the DB results are already merged back. Sandbox
~/tmp/dj-clap-poc/ removable on fedora-lab too (keep dashboard.html +
MISSION-*.md copies anywhere the user wants).