quota_usage.used counter drifts from actual storage

[nikitakodkany]

Problem

GET /projects/{name}/summary returns quota_usage.used which is maintained as a running counter in the database rather than computed from actual blob storage. This counter can drift from reality when:

• An upload fails mid-way
• A transaction rollback does not fully complete

The counter only self-corrects when garbage collection (GC) or an explicit delete operation runs. During the drift window, quota checks are inaccurate.

Current behavior

quota_usage.used is a mutable running counter. It increments/decrements on blob operations but has no reconciliation path for failed or partially rolled back transactions.

Expected behavior

Quota usage reflects actual bytes stored in project_blob at all times, with no drift.

Proposed fix

Harbor already has CalculateTotalSizeByProject internally, which does a live SUM from project_blob. Rather than adding a new endpoint as a workaround, the quota check path should be reworked to eliminate the running counter entirely either by:

1. Live aggregate: Replace quota_usage.used reads with a query against project_blob backed by an appropriate index.
2. Update an indexed aggregate transactionally on every blob insert/delete, eliminating async drift.

Either approach removes the need for GC to act as a correctness mechanism for quota.

Notes

• CalculateTotalSizeByProject exists in the codebase but is not currently exposed via any public API. A GET /projects/{name}/storage endpoint was considered as a short-term workaround but a proper fix to the counter is preferred
• This was discussed in the Harbor community Slack - https://cloud-native.slack.com/archives/CC1E09J6S/p1783454715724819

[Vad1mo]

The counter and the live SUM answer two different questions, and that's the crux.

• SUM(project_blob) answers: how much is stored right now.
• quota_usage.used answers: how much is stored plus reserved by uploads in flight.

Quota enforcement needs the second one. During a push, the layer's bytes have no row in project_blob yet — the association is written after the upload completes. So if the check used a live SUM, N concurrent pushes would each see the project as having room, and all get admitted. Both proposals in the issue (live aggregate, or transactional aggregate on insert/delete) compute the first quantity, so both break enforcement rather than fix drift.

Drift enters through the reservation lifecycle: reserve → upload fails → release fails too (crash, DB error). The reservation is now leaked, and nothing reclaims it until GC recalculates.

So fix the lifecycle, not the data model:

1. On deny, refresh and retry once. A leaked reservation only causes damage when it produces a false "quota exceeded". So when a reservation is denied, refresh usage from project_blob and retry the reservation before returning 403. Push happy path is untouched; the cost only occurs on imminent rejection (with a cooldown so retry storms don't hammer the DB).
2. On failed release, schedule a refresh. Today a failed release is logged and forgotten. Instead, mark the project dirty and refresh it via the existing background task pool. Leaks heal in about a minute instead of at the next GC.

No migration, no new API, no config — both changes are additive and revert cleanly.

Known limitations: the UI value can still be stale between refreshes (this fixes enforcement, not display), and a refresh overwrites concurrent in-flight reservations, so a project at its limit can briefly overshoot — the same window that already exists when a delete triggers a refresh today.

[wy65701436]

Thanks for the detailed writeup @nikitakodkany — the drift you describe is real, but I'd like to add some context on the design before we talk about removing the counter.

quota_usage.used is not just a cached aggregate — it's a reservation ledger. When a client pushes, the quota middleware reserves the size in a row-locked transaction before the bytes are accepted, and the blob only appears in project_blob later (at upload-complete / manifest PUT). A live SUM over project_blob cannot see in-flight uploads, so with concurrent pushes it would allow a project to blow past its limit — the counter is what serializes admission. The same applies to updating an aggregate on blob insert/delete: the insert happens after the bytes are already in. There's also a performance cost to running SUM on the hot push path for large projects.

The drift happens when a reservation leaks. For that, Harbor already has reconciliation beyond GC: the admin API POST /api/internal/syncquota recalculates all project usage from project_blob, and the refresh middleware recalculates after successful delete/manifest operations.

So rather than eliminating the counter, I'd suggest:

1. If you have a reproducible leak path (failed upload / rollback that doesn't release the reservation), please share it — that's a concrete bug we should fix.
2. We could add a scheduled background reconcile (reusing the existing refresh logic) so drift self-heals without waiting for GC or a manual sync.
   That would address the operational pain without changing the enforcement model.