[GSoC] CMRT 01: DB + DAO: Add fingerprint schema and TrackFingerprintDao

[Swarnadip-Kar]

Introduces the three schema revisions and the complete C++ data access
layer that Phase 1 builds on. Everything downstream (Analyzer, Worker,
Queue UI) depends on this PR landing first.

Schema (res/schema.xml, src/database/mixxxdb.cpp)

Revision 41 — adds seven columns to the library table:
musicbrainz_recording_id, musicbrainz_release_id,
musicbrainz_track_id, musicbrainz_artist_id,
acoustid_id, acoustid_lookup_at, acoustid_lookup_status
Plus indexes on musicbrainz_recording_id, acoustid_id,
and musicbrainz_artist_id.

Revision 42 — adds data_text TEXT to track_analysis, with a
composite (track_id, type) index. Used by AnalyzerChromaprint
to store version metadata; raw fingerprint arrays are never
written to the DB.

Revision 43 — five new tables:
fingerprint_metadata — 32-bit SimHash, SHA-256, duration,
version, CMRT group linkage, validity flags, computed_at.
fingerprint_hash is a locality-sensitive pre-filter (NOT
a unique key); full XOR/popcount of .chroma files is always
required to confirm a match. PRIMARY KEY (track_id).
cmrt_groups — one row per unique audio identity. chroma_sha256
is UNIQUE here (one group = one canonical audio identity).
cmrt_members — track-to-group bridge; UNIQUE (track_id) enforces
that each track belongs to at most one group.
acoustid_queue — priority queue for background AcoustID API
submissions. UNIQUE (track_id) makes enqueue idempotent.
acoustid_cache — keyed on chroma_sha256 (SHA-256, not SimHash)
to avoid Birthday Paradox false cache hits.
kRequiredSchemaVersion bumped 40 -> 43.

trackschema.h / trackschema.cpp

Adds string constants for all five new table names and their
columns. tableForColumn() extended to route fingerprint_hash,
chroma_sha256, and fingerprint_duration to fingerprint_metadata,
and group_id / cmrt_groups columns to cmrt_groups.

TrackFingerprintDao (new files)

trackfingerprintdao.h

• DTOs: FingerprintMetadata, CmrtGroup, CmrtMember, AcoustIdJob,
  AcoustIdCacheEntry, UnmatchedTrackInfo.
• Full public API:
  saveFingerprintMetadata / getFingerprintMetadata /
  markFingerprintNeedsRegen / deleteFingerprintMetadata
  createCmrtGroup / getCmrtGroup
  addCmrtMember / getCmrtMembersForGroup / deleteCmrtMember /
  updateCmrtGroupTrackCount
  saveChromaFile / loadChromaFile / deleteChromaFile
  enqueueAcoustId / updateQueueStatus / getPendingJobs /
  deleteQueueEntry
  cacheAcoustIdResult / lookupAcoustIdCache /
  deleteExpiredCacheEntries
  getUnmatchedTracks / reQueueJob
  clearFingerprintData / clearAllFingerprintData
• Private helpers: getFingerprintStoragePath(), getChromaFilePath()
  .chroma files live in ~/.mixxx/fingerprints/track_{id}.chroma

trackfingerprintdao.cpp

• saveFingerprintMetadata(): update-then-insert pattern consistent
  with other Mixxx DAOs; stores computed_at as Unix INTEGER.
• fingerprintHash is quint32 throughout — SimHash is an unsigned
  32-bit value; signed would corrupt pre-filter index comparisons.
• saveChromaFile(): atomic write via .tmp file + rename to prevent
  partial files on crash or disk-full.
• getFingerprintStoragePath(): returns ~/.mixxx/fingerprints/,
  creating the directory on first call if it doesn't exist yet.
• enqueueAcoustId(): uses INSERT OR IGNORE — safe to call
  unconditionally; UNIQUE constraint silently no-ops duplicates.
• clearFingerprintData(): .chroma file removal then
  canonical reassignment or group deletion in cmrt_groups,
  then cmrt_members, fingerprint_metadata, acoustid_queue rows.
• clearAllFingerprintData(): collects all track IDs first to avoid
  mutating the table while iterating over it.

TrackCollection wiring

trackcollection.h / trackcollection.cpp

• m_trackFingerprintDao added as a member (initialized with pConfig).
• Passed into TrackDAO constructor (new fingerprintDao& parameter).
• getTrackFingerprintDAO() accessor added.
• connectDatabase() initializes m_trackFingerprintDao alongside
  the other DAOs.

TrackDAO cascade delete

trackdao.h / trackdao.cpp

• Constructor gains TrackFingerprintDao& fingerprintDao parameter.
• onPurgingTracks() manually DELETEs from fingerprint_metadata,
  cmrt_members, and acoustid_queue for purged track IDs, then
  calls m_fingerprintDao.deleteChromaFile() for each.
  SQLite foreign key constraints are disabled globally in Mixxx
  (see Implementation Note 10.2 in the design doc), so CASCADE
  does not apply — this is the standard Mixxx workaround.
• updateAcoustIdResult() added: UPDATE library SET acoustid_id,
  acoustid_lookup_status, and all four MBID columns for a given
  track. Binds NULL for empty strings to avoid overwriting
  existing data with blanks. Declared as a slot so it can receive
  the worker's cross-thread queued signal.

LibraryScanner

libraryscanner.h

• Adds TrackFingerprintDao m_fingerprintDao member so the scanner
  thread has its own initialized DAO instance.

Notes for reviewers

• fingerprintHash (quint32 / uint32) is intentionally unsigned —
  SimHash bit comparisons on signed integers are undefined behaviour.
• acoustid_cache is keyed on chroma_sha256 (SHA-256), not
  fingerprint_hash (SimHash), to avoid Birthday Paradox collisions.
• Raw fingerprint arrays (.chroma files) are NEVER stored as BLOBs
  in the database — only the SimHash and SHA-256 live in the DB.
• CMRT group assignment (Q1 two-phase SimHash pre-filter +
  XOR/popcount comparison) is deferred to a subsequent PR;
  cmrtGroupId is stored as -1 until then.

[daschuer]

Do you have a relationship graph at hand?

[Swarnadip-Kar]

Yes — the schema was designed upfront in our GSoC collaboration doc.

A few things worth highlighting for the review:

• fingerprint_hash (SimHash) is intentionally non-unique everywhere — it is a pre-filter only. Full comparison of .chroma files is always required before group assignment (In phase 2 I am actually using the % returned in function #15585 )

• chroma_sha256 is UNIQUE only in cmrt_groups (one group = one canonical audio identity), non-unique in fingerprint_metadata (exact audio copies share the same SHA-256)

• acoustid_cache is keyed on SHA-256, not SimHash, to avoid Birthday Paradox false cache hits

• No FK cascade enforcement — SQLite FKs are disabled globally in Mixxx; cleanup is handled manually in onPurgingTracks()

Here's the full ERD:

erDiagram
  LIBRARY {
    int id PK
    varchar title
    varchar artist
    real duration
    varchar acoustid_id "nullable idx"
    int acoustid_lookup_at "Unix ts nullable"
    varchar acoustid_lookup_status "pending|completed|failed"
    varchar musicbrainz_recording_id "nullable idx"
    varchar musicbrainz_release_id "nullable"
    varchar musicbrainz_track_id "nullable"
    varchar musicbrainz_artist_id "nullable idx"
  }
  TRACK_ANALYSIS {
    int track_id FK
    varchar type
    blob data
    text data_text "new: quality metrics JSON" 
  }
  FINGERPRINT_METADATA {
    int track_id PK "FK to library"
    uint32 fingerprint_hash "SimHash NON-UNIQUE pre-filter"
    varchar chroma_sha256 "SHA-256 hex NON-UNIQUE here"
    real fingerprint_duration
    int fingerprint_version
    int cmrt_group_id FK "nullable -1 until grouped"
    real cmrt_offset_seconds
    bool is_canonical
    bool fingerprint_valid
    bool fingerprint_needs_regen
    int computed_at "Unix timestamp"
  }
  CMRT_GROUPS {
    int group_id PK
    uint32 fingerprint_hash "SimHash pre-filter"
    varchar chroma_sha256 "UNIQUE one group per audio identity"
    int canonical_track_id FK "points to library"
    int track_count
    int created_at "Unix timestamp"
    int last_updated "Unix timestamp nullable"
    varchar musicbrainz_cmrt_mbid "nullable idx"
    bool musicbrainz_synced
    int musicbrainz_last_sync "Unix timestamp nullable"
    real musicbrainz_community_score
    bool musicbrainz_submitted
    varchar musicbrainz_submission_mbid "nullable"
    bool local_preferred "default 1"
    int conflict_resolved_at "Unix timestamp nullable"
    varchar conflict_resolution "local_won|remote_won nullable"
  }
CMRT_MEMBERS {
    int member_id PK
    int group_id FK
    int track_id FK "UNIQUE one group per track"
    real offset_from_canonical
    real quality_score "nullable -1.0 sentinel"
    real match_score "nullable -1.0 sentinel"
    bool is_fake_lossless
    int added_at "Unix timestamp"
    int user_quality_rating "nullable -1 sentinel"
  }
  ACOUSTID_QUEUE {
    int queue_id PK
    int track_id FK "UNIQUE idempotent enqueue"
    int priority "lower = higher priority"
    varchar status "queued|processing|completed|failed"
    int attempts
    int max_attempts "default 3"
    int last_attempt "Unix timestamp nullable"
    text error_message "nullable"
    int queued_at "Unix timestamp"
  }
  ACOUSTID_CACHE {
    varchar chroma_sha256 PK "SHA-256 NOT SimHash avoids Birthday Paradox"
    varchar acoustid_id
    varchar musicbrainz_recording_id "nullable idx"
    varchar musicbrainz_release_id "nullable"
    text musicbrainz_metadata "full JSON nullable"
    real confidence "nullable -1.0 sentinel"
    int lookup_timestamp "Unix timestamp"
    int expires_at "Unix timestamp nullable no TTL"
  }

  LIBRARY ||--o| FINGERPRINT_METADATA : "1:0-1 track_id PK"
  LIBRARY ||--o{ TRACK_ANALYSIS : "1:many track_id FK"
  LIBRARY ||--o| ACOUSTID_QUEUE : "1:0-1 UNIQUE track_id"
  LIBRARY ||--o| CMRT_MEMBERS : "1:0-1 UNIQUE track_id"
  CMRT_GROUPS ||--o{ CMRT_MEMBERS : "1:many group_id"
  CMRT_GROUPS }o--|| LIBRARY : "canonical_track_id"
  FINGERPRINT_METADATA }o--o| CMRT_GROUPS : "cmrt_group_id nullable"
  FINGERPRINT_METADATA }o--o| ACOUSTID_CACHE : "chroma_sha256 logical no FK"

Loading