Skip to content

perf(roms): make full rom id index opt-out on GET /api/roms#3781

Merged
gantoine merged 3 commits into
masterfrom
posthog-code/roms-optional-id-index
Jul 17, 2026
Merged

perf(roms): make full rom id index opt-out on GET /api/roms#3781
gantoine merged 3 commits into
masterfrom
posthog-code/roms-optional-id-index

Conversation

@gantoine

Copy link
Copy Markdown
Member

Description
Explain the changes or enhancements you are proposing with this pull request.

Fixes #3770

The home page rails ("recently added" and "continue playing") call GET /api/roms?limit=15&order_by=id&order_dir=desc. That endpoint always computes rom_id_index, the complete ordered list of every matching rom id, which exists to back the gallery's virtual scroll. On a large library (the reporter has ~92,800 roms) this means:

  • a full-table scan (SELECT id FROM roms ORDER BY id DESC) that took ~5.3s uncached, and
  • ~600 KB of JSON (one integer per rom) shipped just to render 15 cards.

There were already opt-outs for the other sidecars (with_char_index, with_filter_values) but none for rom_id_index.

Changes

  • Backend — add a with_rom_id_index query param to GET /api/roms, defaulting to true for backwards compatibility (mirroring the existing sidecar flags). When false, the full index is skipped and the requested page is served directly with offset/limit off the sort index, with a lightweight COUNT(*) for total. Added a get_rom_count() handler helper for the count. Default behavior (virtual-scroll gallery) is unchanged.
  • Frontend — the home rail fetches (getRecentRoms / getRecentPlayedRoms, plus their cache-clear mirrors) now pass with_rom_id_index: false.

A 15-card rail now costs a small page query plus a count, and a few KB of payload.

The response schema is unchanged (rom_id_index is still present, just empty when opted out), so no OpenAPI/type regeneration is needed.

AI assistance disclosure

This change was written with AI assistance (PostHog Code / Claude). The implementation, tests, and verification were AI-generated and human-reviewed.

Checklist
Please check all that apply.

  • I've tested the changes locally
  • I've updated relevant comments
  • I've assigned reviewers for this PR
  • I've added unit tests that cover the changes

Screenshots (if applicable)

N/A (backend/data-fetching change, no visual difference).


Created with PostHog Code

The home rails ("recently added" / "continue playing") request 15 cards
but the endpoint always computed rom_id_index, the complete ordered list
of every matching rom id that backs gallery virtual scroll. On large
libraries this ran a full-table scan and shipped hundreds of KB of ids
to render 15 cards.

Add a with_rom_id_index query param (defaults to true, mirroring the
existing with_char_index / with_filter_values flags). When false, skip
building the index and serve the page directly via offset/limit off the
sort index, with a lightweight count for total. The home rail fetches
pass false.

Fixes #3770

Generated-By: PostHog Code
Task-Id: 45dbad45-bd7c-4cf3-8e8f-105f174f4386
Copilot AI review requested due to automatic review settings July 16, 2026 22:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@greptile-apps

greptile-apps Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR lets small ROM requests skip the full ROM ID index. The main changes are:

  • Adds the backward-compatible with_rom_id_index query parameter.
  • Uses direct pagination and a count when the index is disabled.
  • Disables the index for recent and continue-playing home rails.
  • Keeps matching cache-clear keys and adds endpoint coverage.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.
  • Existing callers retain the original behavior by default.
  • Home-rail consumers only use page items and tolerate the empty index.

Important Files Changed

Filename Overview
backend/endpoints/roms/init.py Adds the opt-out parameter and selects between ID-index pagination and direct database pagination.
backend/handler/database/roms_handler.py Adds a helper that counts the rows represented by the filtered ROM query.
backend/tests/endpoints/roms/test_rom.py Tests the empty index, total, and page contents for an opted-out request.
frontend/src/services/api/rom.ts Disables the full ID index for recent and recently played ROM requests.
frontend/src/services/cache/api.ts Applies the opt-out consistently to cached requests and cache invalidation patterns.

Reviews (1): Last reviewed commit: "perf(roms): make full rom id index opt-o..." | Re-trigger Greptile

greptile-apps[bot]
greptile-apps Bot previously approved these changes Jul 16, 2026
@greptile-apps
greptile-apps Bot dismissed their stale review July 17, 2026 01:01

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@gantoine
gantoine merged commit d760b8d into master Jul 17, 2026
13 checks passed
@gantoine
gantoine deleted the posthog-code/roms-optional-id-index branch July 17, 2026 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Home page rails compute and download a full 92k-entry rom id index to show 15 cards

2 participants