Skip to content

perf(backend): index roms (platform_id, fs_size_bytes)#3780

Merged
gantoine merged 2 commits into
masterfrom
posthog-code/roms-platform-fs-size-index
Jul 17, 2026
Merged

perf(backend): index roms (platform_id, fs_size_bytes)#3780
gantoine merged 2 commits into
masterfrom
posthog-code/roms-platform-fs-size-index

Conversation

@gantoine

Copy link
Copy Markdown
Member

Description

GET /api/platforms (11.5s) and GET /api/stats (~5s) slow down badly as the library grows because both sum roms.fs_size_bytes (per-platform via the Platform.fs_size_bytes column property, and whole-library for stats). fs_size_bytes was not part of any index, so the database had to read actual rows from the very wide, JSON-heavy roms table (964 MB / 92,800 roms in the reporter's case). rom_count on the same model is instant because COUNT is satisfied by the existing platform_id index.

This PR adds a composite index on roms (platform_id, fs_size_bytes), turning both sums into index-only scans that never touch the wide rows. The optimizer picks the covering index for both query shapes on its own, so no query changes are needed.

Changes:

  • backend/models/rom.py: add Index("idx_roms_platform_fs_size", "platform_id", "fs_size_bytes") to Rom.__table_args__.
  • backend/alembic/versions/0097_roms_platform_fs_size_index.py: migration creating the index, using batch_alter_table + if_not_exists/if_exists for MariaDB/MySQL/PostgreSQL compatibility (mirrors the existing 0093 index migration).

Reported measurements (from the issue and a member's replica of a 92,800-rom production DB):

Query Before After
Per-platform SUM(fs_size_bytes) + COUNT (/api/platforms shape) 5.15s 0.07s
Whole-library SUM(fs_size_bytes) (/api/stats shape) 4.06s 0.03s

Index build took ~4.7s on the 964 MB table, so the migration is cheap even for large libraries.

Fixes #3769

AI-assistance disclosure: This change was written with AI assistance (PostHog Code / Claude). The diagnosis, index design, and migration were AI-generated; the timings in the issue and above were verified by a maintainer with read-only queries against a production-scale database.

Checklist

  • I've tested the changes locally
  • I've updated relevant comments
  • I've assigned reviewers for this PR
  • I've added unit tests that cover the changes

Screenshots (if applicable)

N/A (index-only change; see timing table above)


Created with PostHog Code

GET /api/platforms and GET /api/stats both sum roms.fs_size_bytes but the
column was not in any index, forcing full reads of the wide, JSON-heavy roms
table (5-11s on large libraries). A composite index on
(platform_id, fs_size_bytes) turns both sums into index-only scans.

Fixes #3769

Generated-By: PostHog Code
Task-Id: e24ff248-6329-4b70-a826-9a9fee21411e
Copilot AI review requested due to automatic review settings July 16, 2026 22:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@greptile-apps

greptile-apps Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a covering index for ROM size aggregation. The main changes are:

  • Adds an Alembic migration for roms(platform_id, fs_size_bytes).
  • Declares the same composite index in the ROM model.
  • Provides reversible, idempotent index operations.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.
  • The revision chain and migration API match existing repository patterns.
  • The model and migration define the same index.

Important Files Changed

Filename Overview
backend/alembic/versions/0097_roms_platform_fs_size_index.py Adds a reversible composite-index migration using the same batch-operation pattern as existing migrations.
backend/models/rom.py Adds the matching composite index to SQLAlchemy metadata.

Reviews (1): Last reviewed commit: "perf(backend): index roms (platform_id, ..." | Re-trigger Greptile

@gantoine
gantoine merged commit 07bf935 into master Jul 17, 2026
10 of 13 checks passed
@gantoine
gantoine deleted the posthog-code/roms-platform-fs-size-index branch July 17, 2026 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] /api/platforms and /api/stats take 5-11 seconds on large libraries (unindexed SUM of fs_size_bytes)

2 participants