Preserve completed GraphRAG extraction across failed replacements

Fence validated chunk checkpoints in Postgres, retain cancellation drains, and persist measured extraction progress. Restore saved schema read-only with draft-safe UI ordering and qualify semantic estimates against the actual extraction contract.

@codex review

Author: DMontgomery40

docs/exec-plans/active/graphrag-extraction-recovery-2026-09-05.md

@@ -1,6 +1,6 @@
 # GraphRAG Extraction Recovery Implementation Plan
 
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking.
 
 **Goal:** Preserve validated semantic extraction work across failed replacements and show accurate chunk progress without changing graph generation or promotion semantics.
 
@@ -44,12 +44,12 @@ async def prepare_graph_extraction_checkpoint_file(
 ) -> None: ...
 ```
 
-- [ ] Add real Postgres tests first: idempotent duplicate write, conflicting duplicate content, wrong corpus/key, malformed payload, absent/replaced run fence, corpus deletion, explicit deindex, and failed staging cleanup retention.
-- [ ] Run that suite on the private LXC Postgres and retain the behavioral failure evidence.
-- [ ] Add `graph_extraction_checkpoints` with `(repo_id, cache_key)` primary key, a corpus FK with cascade deletion, recipe/file lookup fields, creation time, and JSONB envelope. Register it in corpus-owned deletion inventory.
-- [ ] Canonicalize identity to stable sorted JSON and SHA-256. Validate the envelope and official graph on reads. Check current base-corpus fence inside the same transaction for writes and pruning. Reject conflicting rows; preserve identical rows.
-- [ ] Keep only the current recipe partition and prune obsolete keys only after a file's full current chunk set is known. Successful completion removes deleted-file entries; failed staging reclamation does not remove matching base-corpus entries.
-- [ ] Run the real persistence/lifecycle matrix and Ruff/mypy; independently review transaction ownership and deletion races before integration.
+- [x] Add real Postgres tests first: idempotent duplicate write, conflicting duplicate content, wrong corpus/key, malformed payload, absent/replaced run fence, corpus deletion, explicit deindex, and failed staging cleanup retention.
+- [x] Run that suite on the private LXC Postgres and retain the behavioral failure evidence.
+- [x] Add `graph_extraction_checkpoints` with `(repo_id, cache_key)` primary key, a corpus FK with cascade deletion, recipe/file lookup fields, creation time, and JSONB envelope. Register it in corpus-owned deletion inventory.
+- [x] Canonicalize identity to stable sorted JSON and SHA-256. Validate the envelope and official graph on reads. Check current base-corpus fence inside the same transaction for writes and pruning. Reject conflicting rows; preserve identical rows.
+- [x] Keep only the current recipe partition and prune obsolete keys only after a file's full current chunk set is known. Successful completion removes deleted-file entries; failed staging reclamation does not remove matching base-corpus entries.
+- [x] Run the real persistence/lifecycle matrix and Ruff/mypy; independently review transaction ownership and deletion races before integration.
 
 The identity model must carry explicit fields for the global constraints above. Avoid opaque caller-provided hashes without retaining their typed provenance. The envelope additionally records its originating run and pruning counts; its graph contains no staging/run/corpus scope properties. SQL identifiers are fixed, values parameterized.
 
@@ -59,28 +59,61 @@ The identity model must carry explicit fields for the global constraints above.
 
 **Interfaces:** Build an execution-scoped checkpoint context from the base corpus, owner run, Postgres client, exact approved extraction recipe, file and chunk provenance. Pass it explicitly to the existing extractor. `_DrainingExtractor.extract_for_chunk` consumes that context; the public official return type stays `Neo4jGraph`.
 
-- [ ] Add a real HTTP fixture with several successful domain graphs followed by a timeout/refusal. Restart a separate worker against the same Postgres database and assert that only missing chunk identities are dispatched.
-- [ ] Prove each changed identity input prevents reuse, while changing only run ID, timeout or concurrency preserves reuse. Include two corpora with otherwise identical input.
-- [ ] Inside `extract_for_chunk` (already admitted by the official semaphore), validate a matching stored envelope. Return a deep copy on a hit. A miss calls the existing official method exactly once; SDK and wrapper retries stay disabled.
-- [ ] Validate reserved scope properties, entity-ID consistency and the official graph shape. Apply official `GraphPruning` before persistence, preserving pruning counts. Valid empty graphs can be reused but cannot bypass generation-level nonempty invariants.
-- [ ] Persist before counting reusable success. Retain and drain an initiated commit task on cancellation using the existing producer/drain ownership. Never release the fence while a checkpoint writer still owns work.
-- [ ] Pass the returned copy through unchanged official lexical postprocessing and final file pipeline. Assert that reused graphs acquire only the new run/staging scope and retain document, `NEXT_CHUNK`, and source links.
-- [ ] Exercise cancellation during commit, abrupt worker exit, fence takeover, corrupt rows and storage outage with real components. Failure must not silently issue a replacement paid call.
+- [x] Add a real HTTP fixture with several successful domain graphs followed by a timeout/refusal. Restart a separate worker against the same Postgres database and assert that only missing chunk identities are dispatched.
+- [x] Prove each changed identity input prevents reuse, while changing only run ID, timeout or concurrency preserves reuse. Include two corpora with otherwise identical input.
+- [x] Inside `extract_for_chunk` (already admitted by the official semaphore), validate a matching stored envelope. Return a deep copy on a hit. A miss calls the existing official method exactly once; SDK and wrapper retries stay disabled.
+- [x] Validate reserved scope properties, entity-ID consistency and the official graph shape. Apply official `GraphPruning` before persistence, preserving pruning counts. Valid empty graphs can be reused but cannot bypass generation-level nonempty invariants.
+- [x] Persist before counting reusable success. Retain and drain an initiated commit task on cancellation using the existing producer/drain ownership. Never release the fence while a checkpoint writer still owns work.
+- [x] Pass the returned copy through unchanged official lexical postprocessing and final file pipeline. Assert that reused graphs acquire only the new run/staging scope and retain document, `NEXT_CHUNK`, and source links.
+- [x] Exercise cancellation during commit, abrupt worker exit, fence takeover, corrupt rows and storage outage with real components. Failure must not silently issue a replacement paid call.
 
 ## Task 3: Accurate persisted progress and operator acceptance
 
 **Files:** Modify `server/models/index.py`, `server/api/index.py`, `server/indexing/run_records.py`, the existing Indexing UI graph summary, its browser fixture/spec, and generated contracts. Keep D13 estimate helpers under their existing owner's control.
 
 **Interfaces:** Extend `GraphExtractionTelemetry` with explicit reuse/cancellation/unfinished counts, defaulting new fields for historical records. A typed progress callback publishes admitted/completed outcomes from the extractor into the existing locked run-summary update path. Existing native census and costs remain independent.
 
-- [ ] Add state-transition tests covering queued, admitted, validated, checkpointed, reused, failed and cancelled chunks. Initial queued tasks must not all count as attempted.
-- [ ] Remove semantic-path whole-file `attempted += len(chunks)` and `failed += len(chunks)` on exception. Preserve separate writer/promotion failure evidence and completed extraction counts.
-- [ ] Count success after durable acceptance, reuse separately, and worker duration on actual misses. Provider-latency estimates must exclude reused successes from their denominator.
-- [ ] Persist progress through existing run-record locks without clobbering later accounting or newer progress. Verify deliberately delayed progress/accounting writes in both orders.
-- [ ] Update the Indexing UI to show selected, attempted, reusable successes, reused, failed and unfinished work truthfully after reload. A failed run remains unpromoted even with successful checkpoint rows.
+- [x] Add state-transition tests covering queued, admitted, validated, checkpointed, reused, failed and cancelled chunks. Initial queued tasks must not all count as attempted.
+- [x] Remove semantic-path whole-file `attempted += len(chunks)` and `failed += len(chunks)` on exception. Preserve separate writer/promotion failure evidence and completed extraction counts.
+- [x] Count success after durable acceptance, reuse separately, and worker duration on actual misses. Provider-latency estimates must exclude reused successes from their denominator.
+- [x] Persist progress through existing run-record locks without clobbering later accounting or newer progress. Verify deliberately delayed progress/accounting writes in both orders.
+- [x] Distinguish historical whole-file extraction aggregates from the new per-chunk outcomes. Do not relabel the failed NASA run's old 1,002 attempted/failed values as measured dispatches; retain known selected/chunk totals and show unavailable detailed outcomes for older records.
+- [x] Update the Indexing UI to show selected, attempted, reusable successes, reused, failed and unfinished work truthfully after reload. A failed run remains unpromoted even with successful checkpoint rows.
+- [x] Restore an existing current schema proposal through a typed read-only load after reload. Never generate on mount; missing/stale proposals remain explicit. Preserve corpus/config response ordering and verify the live collapsed review remains usable.
 - [ ] Regenerate TypeScript and contract bundles on LXC100; run changed-surface tests, Ruff, mypy, banned patterns, type/contract/config checks, full CI and the real browser matrix.
 - [ ] Obtain independent Sol xhigh review, complete the PR loop and deployed-marker verification. Only then change NASA's existing per-chunk timeout to a reviewed value, show the corrected estimate, run the approved replacement, and verify domain retrieval/source provenance in the real browser.
 
 ## Self-review and acceptance evidence
 
 The three tasks cover persistence, official-pipeline reuse and operator truth separately. Identity mismatch, stale-writer races, corruption, deletion and cancellation are explicit requirements, not fallback cases. Store interfaces above are shared contracts; rename them coherently if implementation evidence requires a change. Existing native charges for the failed NASA run remain $3.0555729 with incomplete request coverage; this plan cannot recover outputs already lost by that run.
+
+## September 5 implementation verification
+
+Tasks 1 and 2 are implemented and independently approved, including real Postgres
+fence/deletion races, process interruption, exact recipe reuse, retained commits,
+and failed-replacement retention. The combined extraction/storage/recipe family
+passed 578 cases. D13 estimate corrections passed 374 cases and independent review.
+
+Task 3 now persists measured chunk outcomes, captures bounded immutable source
+bytes, and restores saved schema through a read-only endpoint. Schema browser
+acceptance passed 17 cases, including draft Apply/Discard/Generate ordering and
+held authentic responses. Accounting, dock and narrow-screen checks passed;
+the final accounting repeat also passed with deliberately invalid inherited
+Postgres overrides. Three environment-isolation unit tests and independent review
+confirm those overrides cannot redirect the private fixture child.
+
+The full combined backend run passed 3,940 cases with 39 skips. Its only failure
+was the archive fixture lacking Git metadata required by a tracked-file scan.
+After restoring the exact tracked-file index, the clean-start family passed as
+part of the final 79-case progress/index-integration gate. That gate also covers
+late cancellation during a blocked progress write, including coincident write
+failure and repeated cancellation. The old source reproduced two failures in the
+12-case cancellation matrix; the correction and its separate outcome precedence
+passed independent review. Extraction failure stops the pipeline before entity
+resolution, so the run does not invent an additional resolution failure.
+
+Generated types/contracts, configuration checks, Ruff, mypy, web lint and the
+frontend build pass on LXC100. Full CI, publication, deployment, live schema reload,
+and the approved NASA replacement remain open. The earlier failed NASA run is
+still historical aggregate telemetry, and its lost outputs are not recoverable
+from the new checkpoint mechanism.