Document RocksDB backup & restore operations - #590
Conversation
Covers the new backup/restore Operations API operations (create_backup, list_backups, verify_backup, delete_backup, purge_backups, restore_backup) and RocksDB support for get_backup (gzipped-by-default tar; gzip=false for plain tar; remote target support). Documents when a database can be restored online vs offline (system db and component-held dbs require the server stopped), the backupPath storage option, and the 5.2 release note. Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…perations # Conflicts: # release-notes/v5-lincoln/5.2.md
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
…_count) Matches HarperFast/harper#1831, which now returns snake_case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
Backups now has its own sidebar section with an Overview (how the system works, limitations, worked examples for incremental backup/restore and get_backup snapshot download + manual restore) and an Operations page (all seven operations with Operations API and CLI examples). - Add EngineBadge component (globally registered, like VersionBadge) to tag storage-engine support: managed ops are RocksDB-only, get_backup supports RocksDB and LMDB - Remove the Backup Commands section from the CLI commands page; leave a pointer next to the volume-snapshot guidance - Trim operations-api Backup & Restore to the brief index pattern used by Logs/Certificate Management, linking to the new section - Add the backup operations to the CLI operations table - Drop the hyphenated-alias mention entirely Cross-model review (Codex) fixes: scope the whole-database claim and super_user requirement, document the single-root-store restriction, and point LMDB manual restore at storage.path rather than the default path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
1 similar comment
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
- overview: split manual snapshot restore into RocksDB (tar) and LMDB (.mdb) examples; mark the RocksDB walkthrough as engine-specific - cli/commands: frame "How Backups Work" as a per-engine choice (volume snapshots for LMDB, RocksDB backup engine for RocksDB) and scope the atomic-snapshot requirement to LMDB - operations-api: move Backup & Restore section down into the operational cluster (after System, before Jobs) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
| - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. | ||
| - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. | ||
| - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. | ||
| - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. |
There was a problem hiding this comment.
Medium: no documented recovery step for a crashed/interrupted restore
restore_backup is destructive (purge-then-copy). If it's interrupted partway — process crash, power loss, disk error — the implementation leaves a .restoring marker next to the database directory and refuses to load that database on the next harper start (logged as an error: "Incomplete restore ... not loading it — rerun the restore to recover"). None of this is mentioned here or in operations.md: a reader who hits this after an interrupted restore has no way to know from the docs that the fix is simply to rerun restore_backup (same backup_id), not that data is lost or the install is broken.
Suggested fix: add a bullet to Limitations (or a short note under "When can a database be restored?"), e.g. "If restore_backup is interrupted before completing (crash, power loss), the affected database is left unloadable — Harper logs an incomplete-restore error and skips it on startup. Rerun restore_backup for the same database (and backup_id, if you passed one) to recover; do not attempt to load or manually repair the directory."
—
Generated by Barber AI
|
|
||
| Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) | ||
|
|
||
| From the CLI with the server stopped, `target_database=<name>` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start. |
There was a problem hiding this comment.
Low: target_database constraint is stated more strictly than the implementation
"The target must not already exist" isn't quite what restoreBackupOffline enforces — it accepts an existing empty directory too (isMissingOrEmptyDir), rejecting only a non-empty one. Not harmful (following the doc's advice always works), but a reader with an empty leftover directory under that name would incorrectly conclude they need to pick a different name.
Suggested fix: "The target must not already exist (or must be empty); Harper picks the new database up on the next start."
—
Generated by Barber AI
Ethan-Arrowood
left a comment
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
I re-verified @kriszyp's five inline comments against harper#1831's current head — all five still stand, so this needs another round regardless. On top of those, one finding of my own that I'd treat as the blocker: blobs are undocumented end to end, and the snapshot-restore example silently loses blob data for any database with file-backed blobs.
get_backup's tar now carries blobs/<rootIndex>/<relpath> entries. The documented tar -xzf … -C ~/hdb/database/data extracts those inside the RocksDB directory, restoring no blobs to their real roots — a user follows the docs, gets a clean-looking restore, and has lost blob data. exclude_blobs, blob snapshots in managed backups, and blob purge-and-rewrite on restore are all missing, and the "all tables plus the audit/transaction log" scope line plus the 5.2 release-notes entry need the blob carve-in too.
One process note: your CHANGES_REQUESTED summary on this PR says "This looks good" — I think that landed on the wrong PR, since the inline comments are substantive. Either way a re-request will be needed to clear the state.
And the ordering question: #1831's blob work is itself under CHANGES_REQUESTED from kriszyp. These docs should re-sync once that settles — is a final drift pass planned before merge?
sent with Claude Opus 5
…emantics) Address PR #590 review feedback (kriszyp, Ethan-Arrowood), verified against harper#1831 head: - Blobs (blocker): document that get_backup's tar and managed backups include file-backed blobs; add a correct manual snapshot-restore procedure that restores blob roots separately (the naive single-extract recipe silently lost blob data); cover exclude_blobs and restore's blob purge-and-rewrite; carve blobs into the scope line and 5.2 notes - DR copy: backups share files across IDs — copy the whole per-database repository while quiesced, not a single backup dir - verify_backup: scope to RocksDB files + tx-log framing; blobs unverified - Incremental cost: tx-log and blob snapshots are copied in full each backup (not incremental) - restore_backup: online open-DB failure is job status ERROR (not HTTP 409); system/target_database rejected synchronously; target may be an empty dir; document interrupted-restore recovery (rerun restore_backup) - list_backups: define timestamp/size/file_count and note they exclude tx-log/blob snapshots - get_backup: note it can clone a database from another node via target Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks @kriszyp and @Ethan-Arrowood — this was a genuinely useful round. All inline comments are addressed in b2d6053, verified against
One thing worth a second look: against #1831's current head the transaction-log snapshot is verified (a framing check, always), so the only unverified part is the blob snapshot. The docs now reflect that rather than the earlier "tx-log is never checked" framing — please confirm that matches your reading of On the ordering question: this pass is the re-sync against #1831's reworked blob handling. Since #1831 is still moving, I'll do a final drift check against its head just before merge. Re-requesting review to clear the state. — Generated by Barber AI |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
3 participants
Summary
Documents the RocksDB backup & restore feature shipping in 5.2 (companion to HarperFast/harper#1831), in a dedicated Backups reference section:
reference/backups/overview.md) — how the backup system works (incremental managed backups, snapshot download), its limitations (engine support, granularity, single storage root, online-vs-offline restore matrix), and two worked examples: incremental backup/restore withharper create_backup/harper restore_backup, andharper get_backupstreaming atar.gzoff-host plus the manual restore procedure (stop Harper, extract into the database directory understorage.path).reference/backups/operations.md) — all seven operations (create_backup,list_backups,verify_backup,delete_backup,purge_backups,restore_backup,get_backup) with Operations API and CLI examples.src/components/EngineBadge.tsx, globally registered likeVersionBadge) tags storage-engine support per operation: the managed operations are RocksDB-only;get_backupsupports both RocksDB and LMDB.reference/operations-api/operations.md) — Backup & Restore trimmed to the brief index pattern used by Logs/Certificate Management, linking to the new section.reference/cli/commands.md) — the Backup Commands section is removed (content lives in Backups now); the backup operations are listed in the CLI operations table (reference/cli/operations-api-commands.md).reference/configuration/options.md) — thebackupPathstorage option (default<rootPath>/backup).release-notes/v5-lincoln/5.2.md) — Backup & Restore entry, linking to the new section.Where to look
<storage.path>/<database>(default~/hdb/database/data) — path verified againstDATABASES_DIR_NAMEin core and rocksdb-js's tar layout (entries unpack directly into a directory that opens as a RocksDB database, includingtransaction_logs/). Worth a second pair of eyes since a wrong path here damages user data.get_backuppartial exports, the single-root-store restriction is documented as a limitation, thesuper_userrequirement is scoped to server-invoked operations (offline CLI is filesystem-governed), and LMDB manual restore points atstorage.pathrather than the default-only path. The Gemini review leg failed to run (headless permission denial), so outside-model coverage is Codex-only.engines="RocksDB, LMDB") rendered in the same style as VersionBadge — open to a different design.Reviewers not assigned — leaving that to you.
Generated by an LLM (Claude Fable 5).
🤖 Generated with Claude Code