Use persistent staging for Apple Health imports

Author: dzarlax

AGENTS.md

@@ -60,7 +60,7 @@ Single binary HTTP server (`cmd/server/main.go`) that wires together several pac
   ```
   `MaybeFireAll` is called from the morning scheduler tick (see `runMorningSmartRetry` in `cmd/server/main.go`). It honours per-rule cadence + eligibility and never bubbles errors — a misbehaving rule cannot block the morning report path. Render returning `""` is treated as "skip with no persist" (idiomatic when the render itself re-evaluates a condition). See `digest.go` and `energy_nudge.go` for production examples.
 
-- **`internal/applehealth`** — streaming XML parser for Apple Health export files (`export.xml` or `.zip`). Memory-efficient, maps 100+ HK metric types to internal metric names. Normalizes fraction-based percentage metrics (SpO₂, body fat, etc.) to 0–100 scale during import. Admin UI and CLI XML imports stage through `storage.ImportSession`: one logical `import_runs` row + one `health_records` row per import, `pgx.CopyFrom` temp staging, coverage metadata, set-based promote, and one cache rebuild after commit. XML snapshot freshness is time-based using `HealthData@exportDate` when available: XML supersedes older live/HAE exact rows, later live/mobile exact rows remain fresher, and a later XML snapshot may supersede them again. Do not reintroduce broad `RemoveAutoExportForRange` cleanup for XML imports.
+- **`internal/applehealth`** — streaming XML parser for Apple Health export files (`export.xml` or `.zip`). Memory-efficient, maps 100+ HK metric types to internal metric names. Normalizes fraction-based percentage metrics (SpO₂, body fat, etc.) to 0–100 scale during import. Admin UI and CLI XML imports stage through `storage.ImportSession`: one logical `import_runs` row + one `health_records` row per import, `pgx.CopyFrom` into persistent staging tables keyed by `import_run_id`, coverage metadata, set-based promote in a short final transaction, staging cleanup, and one cache rebuild after commit. XML snapshot freshness is time-based using `HealthData@exportDate` when available: XML supersedes older live/HAE exact rows, later live/mobile exact rows remain fresher, and a later XML snapshot may supersede them again. Do not reintroduce broad `RemoveAutoExportForRange` cleanup for XML imports.
 
 - **`cmd/backfill`** — standalone CLI to rebuild caches. Flags: `--force` / `-f`.
 

CLAUDE.md

@@ -60,7 +60,7 @@ Single binary HTTP server (`cmd/server/main.go`) that wires together several pac
   ```
   `MaybeFireAll` is called from the morning scheduler tick (see `runMorningSmartRetry` in `cmd/server/main.go`). It honours per-rule cadence + eligibility and never bubbles errors — a misbehaving rule cannot block the morning report path. Render returning `""` is treated as "skip with no persist" (idiomatic when the render itself re-evaluates a condition). See `digest.go` and `energy_nudge.go` for production examples.
 
-- **`internal/applehealth`** — streaming XML parser for Apple Health export files (`export.xml` or `.zip`). Memory-efficient, maps 100+ HK metric types to internal metric names. Normalizes fraction-based percentage metrics (SpO₂, body fat, etc.) to 0–100 scale during import. Admin UI and CLI XML imports stage through `storage.ImportSession`: one logical `import_runs` row + one `health_records` row per import, `pgx.CopyFrom` temp staging, coverage metadata, set-based promote, and one cache rebuild after commit. XML snapshot freshness is time-based using `HealthData@exportDate` when available: XML supersedes older live/HAE exact rows, later live/mobile exact rows remain fresher, and a later XML snapshot may supersede them again. Do not reintroduce broad `RemoveAutoExportForRange` cleanup for XML imports.
+- **`internal/applehealth`** — streaming XML parser for Apple Health export files (`export.xml` or `.zip`). Memory-efficient, maps 100+ HK metric types to internal metric names. Normalizes fraction-based percentage metrics (SpO₂, body fat, etc.) to 0–100 scale during import. Admin UI and CLI XML imports stage through `storage.ImportSession`: one logical `import_runs` row + one `health_records` row per import, `pgx.CopyFrom` into persistent staging tables keyed by `import_run_id`, coverage metadata, set-based promote in a short final transaction, staging cleanup, and one cache rebuild after commit. XML snapshot freshness is time-based using `HealthData@exportDate` when available: XML supersedes older live/HAE exact rows, later live/mobile exact rows remain fresher, and a later XML snapshot may supersede them again. Do not reintroduce broad `RemoveAutoExportForRange` cleanup for XML imports.
 
 - **`cmd/backfill`** — standalone CLI to rebuild caches. Flags: `--force` / `-f`.
 

README.md

@@ -411,11 +411,11 @@ Admin UI and CLI imports share the same staged storage path:
 
 1. create one logical `import_runs` row and one `health_records` row for the export;
 2. read `HealthData@exportDate` as the snapshot freshness timestamp, falling back to import start only when it is unavailable;
-3. stream parsed metric points and workouts into temporary staging tables with bulk database operations;
-4. promote staged rows in one final transaction, recording coverage metadata and provenance;
+3. stream parsed metric points and workouts into persistent staging tables keyed by `import_run_id` with bulk database operations, without holding a transaction for the full XML parse;
+4. promote staged rows in one final transaction, recording coverage metadata and provenance, then clean the staging rows for that run;
 5. invalidate affected aggregate rows and run one cache rebuild after commit.
 
-If XML parsing or staging fails, the import is marked failed and no cleanup/backfill is run. The old row-by-row `BulkInsertPoints` path is retained only for narrow compatibility helpers.
+If XML parsing or staging fails, the import is marked failed with parsed/staged counters, staging rows for that run are cleaned, and no aggregate cleanup/backfill is run. Startup and new import setup also clean staging rows from failed/committed runs and only clearly abandoned `running` imports. The old row-by-row `BulkInsertPoints` path is retained only for narrow compatibility helpers.
 
 After import, run `make energy-backfill` (CLI) or open **Settings → Historical EnergyBank** (web UI) to compute retrospective EnergyBank snapshots from the imported daily scores. This unlocks per-user verdict band calibration; without it, the cold-start defaults are used until the live orchestrator has accumulated 30+ days of snapshots on its own.
 

docs/ai-plans/2026-07-03-healthkit-import-dedup-batching.html

@@ -251,7 +251,7 @@ <h2>Open Questions</h2>
   <section class="ok">
     <h2>Implemented Changes</h2>
     <ul>
-      <li>Added <code>storage.ImportSession</code> for Apple Health XML imports: durable <code>import_runs</code>, one logical <code>health_records</code> row, temp staging tables, <code>pgx.CopyFrom</code>, coverage rows, and set-based promote.</li>
+      <li>Added <code>storage.ImportSession</code> for Apple Health XML imports: durable <code>import_runs</code>, one logical <code>health_records</code> row, staged <code>pgx.CopyFrom</code>, coverage rows, and set-based promote. The initial temp staging was later replaced by persistent <code>import_run_id</code>-keyed staging in the separate follow-up branch.</li>
       <li>Added provenance columns for <code>metric_points</code> and <code>workouts</code>, plus startup creation of <code>import_runs</code> and <code>import_run_coverage</code>.</li>
       <li>Changed live <code>InsertPoints</code> and <code>UpsertWorkout</code> conflict updates to restore <code>origin='live'</code> and clear <code>import_run_id</code>, so later live/mobile data stays fresher than an older XML snapshot.</li>
       <li>Implemented XML point promote with staging deduplication, conservative stale-row cleanup by staged coverage group, and conflict updates guarded by <code>received_at &lt;= snapshot_at</code>.</li>
@@ -279,7 +279,7 @@ <h2>Verification Run</h2>
     <h2>Known Limitations After Implementation</h2>
     <ul>
       <li>XML deletion remains coverage-group conservative because Apple Health XML sample-level stable IDs are not persisted for metric points.</li>
-      <li><strong>P2 follow-up:</strong> large production imports still hold a staging transaction during parse. If this is too heavy in real use, move to persistent staging keyed by <code>import_run_id</code> and keep only promote transactional.</li>
+      <li><strong>P2 follow-up resolved in separate branch:</strong> persistent staging keyed by <code>import_run_id</code> was implemented in <code>codex/persistent-import-staging</code>, so large production imports no longer hold a staging transaction during the full XML parse; only final promote remains transactional.</li>
       <li>Existing July 2026 XML-imported rows are not retroactively backfilled with <code>origin='apple_health_xml'</code>. Provenance starts with imports performed after this change.</li>
     </ul>
   </section>