Skip to content

fix(metadata): identify archives and folder ROMs by the right file - #4146

Open
Spinnich wants to merge 1 commit into
rommapp:masterfrom
Spinnich:fix/archive-member-hash-lookup
Open

fix(metadata): identify archives and folder ROMs by the right file#4146
Spinnich wants to merge 1 commit into
rommapp:masterfrom
Spinnich:fix/archive-member-hash-lookup

Conversation

@Spinnich

@Spinnich Spinnich commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Description
Explain the changes or enhancements you are proposing with this pull request.

Fixes #3742.

Since 4.9.0 an archive's RomFile carries a composite digest streamed across every
entry in the archive (a170649fe, PR #3412). The metadata handlers were never updated
and kept reading their lookup digests straight off that record, so they silently
switched from "the ROM inside the zip" to "a hash of the whole archive". No ROM database
indexes that composite, so any zip with more than one member (arcade/MAME sets, but also
an ordinary ROM zipped next to a readme or patch) stopped matching by hash and fell back
to filename matching. Single-file zips were unaffected, which is why most libraries
never noticed.

This is a re-land of #4059, which implemented exactly this and was closed unmerged on
2026-08-02. The review request on it is still open and no formal review was ever
submitted, so I've kept half A deliberately close to that diff to keep the re-land
legible.

What changed

Which hashes (the 4.9.0 regression). New RomFile.lookup_hashes decides what a ROM
database should be queried with: chd_sha1_hash for CHDs, else the largest
archive_members entry, else the file's own digests. Hasheous and Playmatch both read
it. CHD behaviour for Hasheous is byte-identical to before; only the archive case moves.

Which file (pre-existing, not a 4.9.0 regression). Playmatch picked its file with
next(file for file in files if file.file_size_bytes > 0) — the first file with bytes,
not the largest, with no is_top_level filter. iter_files is a bare os.walk with no
sort and Rom.files has no order_by, so for a multi-disc folder ROM the file Playmatch
was asked about was arbitrary filesystem order and could differ between two machines
scanning the same library. It was also frequently the .m3u playlist, which carries no
game data and can only ever miss.

To be clear about scope: this half is not part of the 4.9.0 regression. 4.8.1 carries
the same three lines verbatim (ca58028eb, "playmatch should just use the first file",
2025-07-10). It's in here because fixing only the hash selection would leave multi-disc
folder ROMs still failing on Playmatch, and the two questions are the same question.
A new _select_lookup_file replaces both copies of that expression (lookup and
suggestion) with the largest top-level file, tie-broken on path.

Worth looking at hardest

  1. _select_lookup_file's is_top_level filter is a behaviour narrowing, not just a
    tie-break. A folder ROM whose files all live in subdirectories (Game/Disc 1/game.bin)
    now gets no Playmatch lookup at all, where before it got one against an arbitrary
    nested file. Hasheous and ScreenScraper already drop those libraries, so this aligns
    the three, but it is a real loss for that layout. Happy to add a last-resort fallback
    to the largest sizeable file if you'd rather keep the old reach.

  2. submit_manual_match_suggestion now contributes the member's digests alongside the
    archive's fileName/fileSize. Harmless for lookups (a hash match outranks
    FileNameAndSize), but the contributed triple is internally inconsistent. Sending the
    member's name and size would be the fuller fix; I left it out of scope here and it's
    worth its own issue. Relatedly, the suggestion no longer falls back to
    rom.md5_hash/rom.sha1_hash when no file qualifies — it just returns. Contributing
    nothing to a public index beats contributing a composite that is a digest of no single
    file.

  3. Playmatch now looks CHDs up by chd_sha1_hash instead of the container digests. This
    is right in principle and matches Hasheous, but Playmatch's CHD behaviour is the one
    leg I couldn't confirm with a live match.

Deliberately unchanged

ScreenScraper keeps sending the archive's own composite digests. Its jeuInfos lookup
falls back to romnom (the filename), which for an arcade zip is the set name, so those
still resolve; moving it would shift match results across many libraries. There's a new
test pinning that, so a future change there has to be a choice rather than a side effect.
RetroAchievements is a separate RAHasher pipeline and is untouched.

No migration, no re-hash. archive_members has been populated since 4.9.0 and
already carries the member digests, so this reads correct data off rows already in the
database.

Correction to the issue body: #3742 claims the arcade fixtures demonstrate the
Playmatch half. They don't — Playmatch does not index MAME sets at all and returns
NoMatch for them on either hash. An ordinary No-Intro ROM zipped with a readme is
what reproduces the Playmatch half; both are covered below.


AI assistance disclosure. This PR was investigated, implemented, and written
primarily by Claude Code — the root-cause analysis, the code, the tests, and this
description. I directed the work, made the scope calls, reviewed every line, and ran
the verification below myself.

Checklist
Please check all that apply.

  • I've tested the changes locally
  • I've updated relevant comments
  • I've assigned reviewers for this PR
  • I've added unit tests that cover the changes
Verification

Live lookups against the real Hasheous and Playmatch APIs, driving the real
FSRomsHandler.get_rom_files() (only RAHasherService.calculate_hash patched out) into
the real lookup_rom:

Fixture md5 sent before md5 sent now Hasheous Playmatch
aerofgt.zip (8 members) c5347ceb… 694853ae… 'Aero Fighters' hasheous 48064 / igdb 3147 (was HTTP 404) NoMatch (expected)
ga2.zip (19 members) c6ed4e03… 0cd4f9af… 'Golden Axe: The Revenge of Death Adder' 51687 / igdb 5565 (was 404) NoMatch (expected)
SMB3 zip (1 member, control) e883e0d4… identical 'Super Mario Bros. 3' 25611 (unchanged) SHA1 igdb 1068 (unchanged)
SMB3 + readme (2 members) 9f018a5b… e883e0d4… 'Super Mario Bros. 3' (was 404) SHA1 igdb 1068 (was NoMatch)

The two arcade NoMatches are not a failure: Playmatch doesn't index MAME on either
hash. The SMB3+readme row is the fixture that demonstrates the Playmatch half.

Multi-disc folder ROM (Metal Gear Solid (USA) (Rev 1)/, two CHDs plus an .m3u):
picks Disc 1, not the playlist; sends the CHD's disc-data SHA-1, not the container's;
identical selection across 4 list orderings (order-dependent on master).

Tests. 15 new tests. All 7 in tests/models/test_rom.py failed on master with
ImportError: cannot import name 'LookupHashes'; the Hasheous payload test failed
sending compositemd5/sha1/crc; 6 of the 7 Playmatch tests failed on master (composite
hashes, fileName='game.m3u', order-dependent payload, a nested file selected, a
composite POSTed as a suggestion). Edge cases covered: single-member archive unchanged,
archive_members None and [], a member with a missing or None size, CHD
precedence, nested-file exclusion, and no-eligible-file.

Full backend suite: 2762 passed, 2 skipped. trunk fmt + trunk check --no-fix on
all 7 files: no issues.

No OpenAPI change — proven, not assumed: dumped app.openapi() from a master
worktree and from this branch and diffed them. Identical, 26,574 lines each. So no
npm run generate and no frontend typecheck.

Screenshots (if applicable)

N/A — backend only. No UI, no i18n, no schema change. Evidence is the live-lookup table
above.

Since 4.9.0 a multi-file archive's RomFile carries a composite digest
streamed across every entry, with per-member digests kept alongside in
archive_members. Hasheous and Playmatch read their lookup digests
straight off that record, but both databases index a multi-file archive
by its largest internal member, so those ROMs stopped matching and fell
back to filename matching.

Add RomFile.lookup_hashes, which decides which digests a ROM database is
queried with: a CHD's embedded disc-data SHA-1, else the largest
archive member, else the file's own. Hasheous and Playmatch both read
it. ScreenScraper deliberately stays on the composite, since its
jeuInfos lookup falls back to romnom.

Playmatch also picked its file with the first one that had bytes,
unfiltered by is_top_level. iter_files is a bare os.walk with no sort
and Rom.files has no order_by, so a multi-disc folder ROM was identified
by an arbitrary file that could differ between machines scanning the
same library. Select the largest top-level file instead, as Hasheous and
ScreenScraper do, with a path tie-break for determinism. This half is
pre-existing rather than a 4.9.0 regression, but leaving it would keep
multi-disc folder ROMs failing on Playmatch.

The manual-match suggestion shares that selection, and no longer falls
back to the ROM-level composite: contributing nothing to a public index
beats contributing a hash of no single file.

Existing libraries need no re-hash; archive_members has been populated
since 4.9.0, so a rescan is enough.

Fixes rommapp#3742

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR centralizes ROM-database lookup hash selection and updates Hasheous and Playmatch to identify archives by their largest member and CHDs by their embedded SHA-1. It also makes Playmatch file selection deterministic for multi-file folder ROMs and adds regression coverage for provider-specific behavior.

  • Adds RomFile.lookup_hashes for CHD, archive-member, and ordinary-file digest selection.
  • Uses those lookup hashes in Hasheous lookups and Playmatch lookups and suggestions.
  • Selects the largest top-level Playmatch file with deterministic path tie-breaking.
  • Adds tests for archives, CHDs, nested files, ordering, empty inputs, suggestions, and unchanged ScreenScraper behavior.

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issues identified.

The shared hash-selection behavior is consistently applied to Hasheous and Playmatch, deterministic file selection removes filesystem-order dependence, and focused tests cover the changed archive, CHD, and folder-ROM paths.

Important Files Changed

Filename Overview
backend/models/rom.py Adds a cached lookup-hash abstraction that prioritizes CHD disc hashes, then the largest archive member, then the file’s own hashes.
backend/handler/metadata/hasheous_handler.py Replaces provider-local CHD handling with the shared lookup-hash selection while retaining empty-hash filtering.
backend/handler/metadata/playmatch_handler.py Makes lookup-file selection deterministic and applies shared lookup hashes consistently to identification and manual suggestions.
backend/tests/models/test_rom.py Covers ordinary files, CHDs, archive-member selection, legacy archive states, and incomplete member sizes.
backend/tests/handler/metadata/test_playmatch_handler.py Covers archive payloads, multi-disc selection, ordering, nested-file exclusion, empty inputs, and suggestion payloads.
backend/tests/handler/metadata/test_ss_handler.py Pins ScreenScraper’s deliberate continued use of composite archive hashes.
backend/tests/handler/test_fastapi.py Verifies Hasheous receives the largest archive member’s hashes.

Reviews (1): Last reviewed commit: "fix(metadata): identify archives and fol..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Hasheous and Playmatch hash matching broken for multi-file archives since 4.9.0 (composite hash sent, but they index by largest file)

1 participant