prerender-v0.15.0 — @harperfast/prerender

Makes the sitemap walk survivable, observable, and unable to hang — sized for a real sitemap index (tens of children, 1M+ target writes) rather than a single HTTP request.

Cannot hang

Every RenderTarget.put also writes a residency-pinned RenderSchedule row, which Harper forwards to the owning node with no deadline of its own. An unreachable owner meant the promise never settled — and .catch() cannot help, because a hang never rejects — so a bulk walk stopped mid-batch with no error and no partial result. util/timeout.js bounds it; a timed-out schedule write degrades to exactly the gap util/reconcile.js already repairs, and is counted and logged.

The schedule delete is bounded too, but propagates rather than degrading: claim builds jobs from the schedule row alone and never checks the target still exists, so an orphaned row would be re-rendered forever.

The same protection covers the bot path and RenderQueue.claim, which awaits its lease writes while holding the queue mutex.

Observable

POST /sitemaps/<url> now answers immediately and walks in the background, persisting progress to SitemapRefresh under the root URL. Poll GET /sitemap_refresh/<root-url> for state, sitemapsProcessed of sitemapsDiscovered, live counts, and an updatedAt bumped after every child — so a stalled walk is distinguishable from a slow one. {"background": false} restores blocking.

A second refresh of a root already running is refused; a run whose progress goes stale past sitemap.staleRunMs is taken over, so a restart mid-walk cannot block a root permanently.

Cannot die on one child, cannot lie

• A child that 503s or returns an HTML error page is recorded in failed[] and the walk continues. Only a failing root propagates.
• removed is an exact count plus a bounded sample, replacing an unbounded array of full records in the HTTP body.
• collectFromScan's truncated is now reported instead of discarded, in both the prune and the delete.
• Caught errors carry their full cause chain — undici reports every network failure as the bare word fetch failed, with the actual reason on cause.

Cheaper and more correct

• The prune scan already proves a key is present and correctly attributed, so it feeds a fast path that removes the per-entry point read entirely in the steady state — previously one sequential read per entry × device, every pass.
• Re-attribution patches instead of re-putting. A put recomputes getInitialRenderTime, so a URL listed in two sitemaps — or shuffled across fixed-size paginated product sitemaps — pushed its own next render forward on every pass.
• parentUrl records which index listed a sitemap, so refresh-all visits roots only. Children were previously walked twice: once via the index, once as top-level rows.
• Deleting an index now cascades. Targets are attributed to the child that listed them, so an index delete previously removed zero targets and stranded every child row.

Notes

• sitemap.background defaults to true, changing the POST response shape. No consumer parses the old shape.
• New config: render.scheduleWriteTimeoutMs (15s), sitemap.background, sitemap.staleRunMs, sitemap.removedSampleCap, sitemap.failedCap.
• New REST surface: sitemap_refresh (authenticated, same posture as the other exports).
• sitemapTargetNeedsUpdate is superseded by actionForExisting and removed.

Verified with 282 unit tests plus an end-to-end smoke test against a live Harper 5.1.15 — see #47 for the full results.