feat(usage-maintenance): complete archive, import, and diagnostics workflow

apps/docs/en/manual/usage-analytics.md

@@ -41,6 +41,10 @@ Filters include model, API key, provider, status, auth file, latency, and cache
 
 If the model name is an alias or internal name, add the matching entry in [Model Prices](./model-prices.md), or cost will be underestimated or empty.
 
+## Export And Historical Boundaries
+
+The complete JSONL export in Usage Maintenance is a stable, ordered snapshot of raw `usage_events` and is independent of `USAGE_QUERY_LIMIT`. It is not a complete CPAMP backup, and history already deleted from the raw table is not automatically merged back from archive segments. For migration or disaster recovery, follow [Backup And Restore](../operations/backup.md) and back up SQLite, WAL/SHM, `data.key`, and `usage-archives/` together.
+
 ## Long Histories And Query Behavior
 
 - Each tab requests only the data it needs; stable filter selectors load separately from main analytics.

apps/docs/en/operations/backup.md

@@ -10,6 +10,7 @@ Back up these files as a set:
 - `usage.sqlite-wal`
 - `usage.sqlite-shm`
 - `data.key`
+- `usage-archives/` when historical archiving has been used
 
 If your deployment directory contains custom configuration files, back them up too. With the one-click installer, also back up `secrets/` and `data/` under the install directory; after a successful import, `secrets/cpa-management-key` is normally gone, but it may remain after a failed upgrade or with `CPAMP_SKIP_EXECUTE=1` for retry. Manual env/secret deployments should back up their matching secret files.
 
@@ -23,6 +24,10 @@ CPA connections saved through setup or the panel encrypt the CPA Management Key
 
 If a CPA connection is managed by manual environment variables or secret files, the CPA Management Key may not be written to SQLite. Back up the related secret files together with the data directory. The installer's env input is migrated into SQLite after success, so do not rely on the one-time input file alone.
 
+Archive files may contain event-level `fail_body` and `raw_json`, so protect them as sensitive data. SQLite, WAL/SHM, `data.key`, and `usage-archives/` must come from the same consistent backup point. With custom `dataDir` or `dbPath` settings, confirm the resolved archive location in the [Manager Server Guide](./manager-server.md); it can be separate from the SQLite directory. Never delete WAL manually or restore only selected archive runs.
+
+The Usage Maintenance “Export usage” action is a complete JSONL snapshot of the current raw `usage_events`. It is independent of `USAGE_QUERY_LIMIT`, streams bounded batches from a stable snapshot boundary, and excludes events written after the export began. It is not a complete CPAMP backup and does not automatically merge raw events already deleted from SQLite back out of `usage-archives/` segments; back up the complete data set above for migration or disaster recovery.
+
 ## Docker Backup Example
 
 If you use a named volume, stop the container first, then export through a temporary container:
@@ -64,12 +69,64 @@ Copy-Item -Recurse .\data .\data.backup
 1. Stop CPAMP.
 2. Restore the full data directory.
 3. Confirm that `usage.sqlite` and `data.key` come from the same backup.
-4. If the CPA connection is env/secret-managed, also restore `secrets/` from the install directory.
-5. Start CPAMP.
-6. Log in and check configuration, monitoring data, and collector status.
+4. When historical archiving has been used, restore the matching `usage-archives/` directory.
+5. If the CPA connection is env/secret-managed, also restore `secrets/` from the install directory.
+6. Start CPAMP.
+7. Log in and check configuration, monitoring data, Usage Maintenance status, and collector status.
 
 If restore produces decryption errors, first check whether `data.key` matches the SQLite database.
 
+## Restore Raw Request History From A Verified Archive
+
+Prefer a complete restore from SQLite, WAL/SHM, `data.key`, and `usage-archives/` captured at the same point in time. The segment-import procedure below is for recovering archived request history into an isolated environment when the original database is unavailable. It is not a table-level merge procedure for a live production database.
+
+Archive segments are `.jsonl.gz` files, while usage import reads decompressed JSONL. Renaming the file is not sufficient: neither the panel file picker nor the import endpoint transparently decompresses gzip. Recover as follows:
+
+1. Use only a run whose status is `verified` or `completed`. Preserve its manifest and original segments; do not edit the archive files in place.
+2. Start an isolated recovery instance with an empty data directory and therefore an empty `usage.sqlite`. Do not import the segments into the source database that still contains the original identity ledger. That database intentionally skips the archived identities, which validates idempotency but does not restore raw rows.
+3. Copy the segments to a restricted scratch directory and decompress them in filename sequence. For example:
+
+```bash
+mkdir -p ./archive-restore
+chmod 700 ./archive-restore
+gzip -dc -- "./usage-archives/<run-id>/<segment-name>.jsonl.gz" \
+  > "./archive-restore/<segment-name>.jsonl"
+chmod 600 "./archive-restore/<segment-name>.jsonl"
+```
+
+On Windows, use a trusted gzip tool to produce the same `.jsonl` file. The decompressed file can still contain `fail_body` and `raw_json`, so continue to handle it as sensitive data.
+
+4. Sign in to the isolated recovery instance and import each decompressed `.jsonl` through Request Monitoring in segment-number order. Do not select the `.jsonl.gz` file directly.
+5. On an empty recovery instance, each segment's `added` count should equal its manifest `event_count`, and `skipped` should be `0`. The sum across all segments should equal the manifest event total. Then verify Request Monitoring, Usage Analytics, and sampled event fields.
+6. Keep the original archive and complete backup until validation is finished. Do not overwrite a live production data directory with the recovery instance's SQLite file, and do not merge tables manually. A complete production rollback must restore the consistent backup set.
+
+If importing the same decompressed segment into the source database reports every event as `skipped`, the identity ledger is correctly preventing archived events from being resurrected. That is an idempotency check, not a failed recovery.
+
+When a browser resumes an import session with an uploaded prefix, it computes the selected file's prefix SHA-256 incrementally and the server compares it with the persisted digest. The selected file must have the same content, not merely the same filename, size, or `lastModified`; a mismatch stops resumable upload and requires a new session. Legacy sessions whose uploaded prefix has no digest are not treated as safe resume targets.
+
+The archive “Abandon task” action is allowed only for `previewed` runs and `failed` runs that have not published any archive segments and have not entered raw deletion. Once any segment has been published (including published `archived`, `verified`, and `failed` runs), or once the run enters `deleting`, partial deletion, or `completed`, cancellation is rejected. Cancelling does not delete raw usage, published archive segments, or identity-ledger entries; a run that has started raw deletion must be resumed or completed.
+
+Note: A complete backup taken strictly before the first raw deletion (consisting of `usage.sqlite`, `usage.sqlite-wal`, `usage.sqlite-shm`, `data.key`, and `usage-archives/`) serves as the recovery boundary if a future upgrade requires complete historical raw events to rebuild derived data. After raw events are deleted, pricing model sets and context-tier thresholds are frozen, and a full historical rebuild may require restoring that pre-deletion backup or using a dedicated migration path provided by that version.
+
+## Reclaim Physical Space After Logical Deletion
+
+Deletion in the Usage Maintenance page removes only archived and verified raw rows. It does not immediately shrink the SQLite file. After completing the stopped backup above, run:
+
+```bash
+cpa-manager-plus compact-usage --db-path ./data/usage.sqlite
+```
+
+For a Docker named volume, run the offline command through the same image:
+
+```bash
+docker compose stop cpa-manager-plus
+docker compose run --rm --no-deps cpa-manager-plus \
+  compact-usage --db-path /data/usage.sqlite
+docker compose up -d cpa-manager-plus
+```
+
+Stop every Manager Server connected to the database before running the command. The process-level database lock rejects a running Manager Server, and SQLite exclusive access rejects conflicting transactions. Any recorded maintenance lock or active `archiving`/`verifying`/`deleting` stage also blocks compaction; static `previewed`, `archived`, `verified`, and `failed` runs are allowed. Pending derived-data migrations are preserved exactly and continue after the server restarts; `compact-usage` does not advance, reset, or rewrite their checkpoints. If a lock belongs to a resumable active or failed run, start Manager Server and resume that run before retrying. If a lock remains for an inactive or terminal run, preserve the backup and logs and stop for diagnosis; never delete the lock or WAL manually. Keep the complete backup and reserve temporary free space conservatively equal to at least the current database-file size. After compaction, start the server and verify `/health`, `/status`, Dashboard, Usage Analytics, and Usage Maintenance. After decompressing one archive sample as described above, re-importing it into the source database should remain an idempotent skip, while importing it into an empty isolated recovery instance should add the event.
+
 ## Move Manager Configuration Without Request History
 
 If the old `usage.sqlite` is large and request history is no longer needed, start the replacement instance with an empty data directory and use the existing Manager configuration API to move the non-sensitive CPA URL, collector, Codex inspection, and External Usage Service settings. This does not copy `usage_events`, rollups, inspection run history, model prices, API Key aliases, or account-processing policy, and it does not export the CPA Management Key.

apps/docs/en/operations/configuration.md

@@ -70,3 +70,14 @@ Constraints:
 - `pollIntervalMs` should not exceed the CPA usage queue retention.
 - CPA retention defaults to 60s and is capped at 3600s.
 - Only one Manager Server should consume the same CPA queue.
+
+## Usage Lifecycle Configuration
+
+Manager Server environment variables control resumable imports and historical retention:
+
+- `USAGE_QUERY_LIMIT` limits compatible Usage queries only; it does not limit the complete raw JSONL export exposed by Usage Maintenance.
+- `USAGE_IMPORT_CHUNK_BYTES` is the server upload chunk size; the panel displays the session's returned `chunk_size_bytes` dynamically.
+- `USAGE_IMPORT_DISK_QUOTA_BYTES`, `USAGE_IMPORT_MAX_SESSIONS`, and `USAGE_IMPORT_SESSION_TTL_MINUTES` control the total temporary-file quota, active-session count, and session TTL.
+- `USAGE_ARCHIVE_RETENTION_ENABLED` and `USAGE_ARCHIVE_RETENTION_DAYS` control automatic retention; retention remains gated by hourly rollup and archive-coverage readiness. When enabled, once archive verification and deletion coverage gates are satisfied, historical raw events older than the retention window are deleted automatically. Note that after the first raw deletion occurs, the set of priced models and the context-tier threshold structure enter a fail-closed freeze and structural mutations are rejected; rate-only updates, including service-tier rates, remain allowed.
+
+The complete export covers only raw usage events still present when the export snapshot starts. It does not merge deleted archive segments back into JSONL and does not replace a backup containing SQLite, WAL/SHM, `data.key`, and `usage-archives/`.