Description
Reconciliation re-fetches every issue and pull request in a repository on every pass, whether or not anything changed. Nothing is cached and nothing is asked incrementally, so the cost of a pass is set by the size of the repository rather than by the amount of activity in it.
The two registered repositories hold 88 and 294 issue rows, and completed passes were starting 20 to 90 seconds apart. Since the per-issue timeline read landed, each pass also spends at least one request per issue. The result is issue 190: the GitHub budget is exhausted, no pass has completed since 15:40:57, and the ledger has stopped updating.
The API is not being asked the question the fold actually has. The fold wants to know what changed since it last looked; it asks instead for the entire current state of every issue and reassembles the answer locally.
Expected Behavior
A reconciliation pass costs an amount of API budget proportional to what changed since the previous pass, not to the size of the repository. A repository with no activity since the last pass costs approximately one request to establish that.
Reproduction Steps
- Let reconciliation run against a repository where nothing has changed since the previous pass.
- Count the GitHub requests it makes. It fetches every issue, every issue's timeline, and the associated pull requests, and writes back rows identical to the ones already stored.
select count(*) from reconciliation_changes where change_kind = 'CHANGE' — 17,000-plus rows, overwhelmingly rewrites of unchanged values, which is the same waste visible from the database side.
Environment / Context
src/lib/github/client.ts performs the fetch and src/lib/fold/reconcile.ts drives it. The store already carries updated_at on issues and pull requests, so the watermark this needs is largely present.
Directly relevant to any design here: the fold currently recomputes every closure from a full snapshot on every run, and several properties depend on that. updateSettlement rewrites every column, calibrations upsert, and closures no longer emitted are deleted. That is why issue 176's repair was expected to fix historical closures with no data migration — a later run simply recomputes them correctly. A cache that stops re-deriving unchanged issues removes that self-healing property, and a defect fixed afterwards would no longer repair rows written while it was present.
Suggested Fix
Unverified. Store a per-repository watermark — the newest updated_at seen in a completed pass — and ask GitHub only for issues updated since it. The REST issues listing takes a since parameter directly; whether the GraphQL issues connection exposes an equivalent filter should be confirmed before the shape is chosen, since the fold is on GraphQL today. Issues absent from that response are unchanged, and their stored rows stand.
The watermark has to be advanced only on a pass that completed, or a failure mid-pass silently skips whatever it had not yet read. Given the queue added in issue 13, a failed job returning to PENDING with the watermark unmoved is the natural shape.
Two things need deciding rather than assuming:
- Whether every event the fold reads moves the issue's
updated_at. Labels and comments are the evidence this system prices, and if any of them can change without moving that timestamp, a watermark silently misses exactly the events that matter most.
- How the self-healing property above is preserved. A periodic full pass, or a way to invalidate the watermark deliberately after a fold change, keeps a future repair able to reach historical rows. Without one, fixing a fold defect would additionally require a migration to re-derive what it corrupted.
Description
Reconciliation re-fetches every issue and pull request in a repository on every pass, whether or not anything changed. Nothing is cached and nothing is asked incrementally, so the cost of a pass is set by the size of the repository rather than by the amount of activity in it.
The two registered repositories hold 88 and 294 issue rows, and completed passes were starting 20 to 90 seconds apart. Since the per-issue timeline read landed, each pass also spends at least one request per issue. The result is issue 190: the GitHub budget is exhausted, no pass has completed since 15:40:57, and the ledger has stopped updating.
The API is not being asked the question the fold actually has. The fold wants to know what changed since it last looked; it asks instead for the entire current state of every issue and reassembles the answer locally.
Expected Behavior
A reconciliation pass costs an amount of API budget proportional to what changed since the previous pass, not to the size of the repository. A repository with no activity since the last pass costs approximately one request to establish that.
Reproduction Steps
select count(*) from reconciliation_changes where change_kind = 'CHANGE'— 17,000-plus rows, overwhelmingly rewrites of unchanged values, which is the same waste visible from the database side.Environment / Context
src/lib/github/client.tsperforms the fetch andsrc/lib/fold/reconcile.tsdrives it. The store already carriesupdated_aton issues and pull requests, so the watermark this needs is largely present.Directly relevant to any design here: the fold currently recomputes every closure from a full snapshot on every run, and several properties depend on that.
updateSettlementrewrites every column, calibrations upsert, and closures no longer emitted are deleted. That is why issue 176's repair was expected to fix historical closures with no data migration — a later run simply recomputes them correctly. A cache that stops re-deriving unchanged issues removes that self-healing property, and a defect fixed afterwards would no longer repair rows written while it was present.Suggested Fix
Unverified. Store a per-repository watermark — the newest
updated_atseen in a completed pass — and ask GitHub only for issues updated since it. The REST issues listing takes asinceparameter directly; whether the GraphQLissuesconnection exposes an equivalent filter should be confirmed before the shape is chosen, since the fold is on GraphQL today. Issues absent from that response are unchanged, and their stored rows stand.The watermark has to be advanced only on a pass that completed, or a failure mid-pass silently skips whatever it had not yet read. Given the queue added in issue 13, a failed job returning to PENDING with the watermark unmoved is the natural shape.
Two things need deciding rather than assuming:
updated_at. Labels and comments are the evidence this system prices, and if any of them can change without moving that timestamp, a watermark silently misses exactly the events that matter most.