Skip to content

CLAUDE.md

Latest commit

 

History

History
220 lines (181 loc) · 14.2 KB

CLAUDE.md

File metadata and controls

220 lines (181 loc) · 14.2 KB
Raw

CLAUDE.md

Guidance for working in this repository.

What this is

deep-freeze is a personal, incremental backup system that uploads encrypted archives to Amazon S3, defaulting to the Glacier Deep Archive storage class (cheapest, slow + costly to restore — assumes restores are rare, e.g. total disk loss). All backup state (what has been backed up, into which archive) lives in a local SQLite3 database. That database is itself backed up (it must be, since Deep Archive can't be read quickly/cheaply).

Language: Python 3 (stdlib only for the core; numpy/matplotlib only for report.py). No package manager beyond requirements.txt. External CLI dependencies: aws (AWS CLI), gpg, and GNU coreutils/tar (see the launchd plist PATH).

Architecture / how a backup runs

Entry point deep-freeze.py → backup.Coordinator.run():

  1. Coordinator.run()/run_manual() take RunLock (backup/lock.py), an exclusive non-blocking flock on ~/.deep-freeze-backups/deep-freeze.lock, for the whole call. If another deep-freeze/deep-freeze-manual.py process already holds it, this raises AlreadyRunning immediately rather than blocking (entry-point scripts catch it and exit 75). Because it's a kernel flock, a holder that dies for any reason drops the lock automatically — there's no stale-lock file to detect or clean up.
  2. ClientConfigFactory.get_active_client_configs() loads every active config from the DB (the config whose backup_root is the deep-freeze dir itself is always processed last, so the DB reflecting this run gets backed up after everything else).
  3. For each config (skipping manual_only ones) → Backup(db, cc).run() then Purge(db, cc).run().

Backup.run() (backup/backup.py) is the core incremental algorithm:

  1. prepare_backup() (db.py, scoped to this config's client_fqdn/backup_root) — clears any pending_upload leftovers from an interrupted prior run of this config and resets its new_* columns.
  2. set_sweep_mark(...) — marks all known files for this config as sweep_mark='absent'.
  3. scan() — os.walk over backup_root. For each non-excluded file, File.upsert() inserts/updates the files row, setting sweep_mark='present' and the new_size / new_modification / new_status='present' staging columns. Directories on other mount points are skipped unless backups_cross_devices=Y.
  4. update_deleted_files_new_status(...) — files still sweep_mark='absent' (i.e. gone from disk, including newly-excluded) get new_status='absent'; their archive records are flagged deleted and the superseded archives' relevant_size is decremented.
  5. mark_files_for_backup(...) — sets force_backup='Y' where size or mtime changed.
  6. Refresh(db, cc).run() (backup/refresh.py) — picks old, partially-irrelevant archives and also sets force_backup='Y' on the files they still hold (see "Archive refresh" below). Must run after scan()/update_deleted_files_new_status() so new_size is populated for anything it marks (see backup/refresh.py for the crash reasoning).
  7. backup() — bundles files needing backup into tar.gz archives, rolling to a new archive at archive_max_size_bytes (500 MB, hardcoded). Each archive is gpg symmetric-encrypted (gpg -c, passphrase from the config's key file) to <name>.enc, uploaded via aws s3 cp --storage-class DEEP_ARCHIVE. Upload failure aborts (check_returncode()). On success archive_uploaded() promotes new_* → live columns, marks records uploaded, and flags prior backups of those files superseded.

Archive object key format (new_archive_name): YYYY/MM/DD/HH-MM-SS_<client_fqdn>_<safe_backup_root>_<seq>.tar.gz.enc — date-sharded to limit per-prefix object counts.

Purge.run() (backup/purge.py): fix_stats() recomputes each archive's relevant_size from its still-uploaded records, then deletes from S3 (aws s3 rm) any archive that is 100% irrelevant (relevant_size = 0) and older than 180 days, flagging it deleted in the DB. Partially-irrelevant archives are handled by Refresh (below), which drives them to 0% relevance so Purge picks them up in the same run — Purge itself is unchanged.

Archive refresh

Refresh.run() (backup/refresh.py) reclaims archives that are past the 180-day minimum charge duration but only partially irrelevant (some files superseded/deleted, some still current) — these would otherwise stick around forever, since Purge only ever deletes at exactly relevant_size = 0. It picks the least-relevant, oldest eligible archives, force-backs-up the files they still hold via db.mark_archive_files_for_backup, and lets the existing supersession/purge machinery retire them. Nothing about tarring, encryption, upload, supersession or purge changes — this only adds a selection step before backup().

  • Eligibility is age-scaled: relevance_ratio = relevant_bytes / total_size (computed live from file_archive_records, not the possibly-stale s3_archives.relevant_size column); an archive is eligible once age_days > 180 + relevance_ratio * 180. A 10%-relevant archive is eligible at ~198 days; a 90%-relevant one waits until ~342 days — highly-relevant archives are deferred, not excluded.
  • Per-run cap, per client config: selection stops (does not skip ahead) at the first candidate that would exceed either a count cap (refresh_max_archives_pct, default 5% of the config's uploaded archives, floored to at least 1) or a byte cap (refresh_max_bytes, default 2 GB of relevant bytes re-uploaded per run).
  • Ordering: least-relevant first, oldest (created) as tiebreaker.
  • Inspect the candidate queue without running a backup: deep-freeze-ctl.py archives refresh-candidates --client-name --backup-root.

The staging-column pattern (important)

The files table carries both live columns (size, modification, status, last_archive_id) and new_* staging columns plus sweep_mark and force_backup. A run writes observations into new_* first; only a successful upload (or confirmed deletion) copies them onto the live columns and nulls the staging ones. This is what makes the process crash-recoverable: an interrupted run leaves live state intact, and the next prepare_backup() cleans the staging area.

Database

SQLite at ~/.deep-freeze-backups/deep-freeze-backups.db (Database in db/db.py, hardcoded path). row_factory = sqlite3.Row. Almost all SQL lives in db/db.py as methods on Database.

Schema migrations

db/ddl.py MaintainSchema runs on every connect. Version is stored in deep_freeze_metadata (key='schema_version'); it applies db/schema_upgrades/v{N}.py classes (SchemaUpgradeV{N}) in order up to target_schema_version (currently 3). To change the schema: add a new vN.py, export it in db/schema_upgrades/__init__.py, and bump target_schema_version. Never edit an existing upgrade file — migrations run against live user DBs.

Tables

deep_freeze_metadata(key PK, value) — key/value; holds schema_version.

backup_client_configs — one row per (client, directory) backup config. PK (client_fqdn, backup_root). Columns: cloud, region, bucket, credentials (= AWS CLI profile name), client_fqdn, backup_root, status (active), key_file_path.

backup_client_configs_options — (option_id PK, client_fqdn, backup_root, key, value), unique on (client_fqdn, backup_root, key). Known keys (constants in ClientConfig): backups_cross_devices, manual_only, temporary_directory; values 'Y'/'N' or a path. Also refresh_archives ('Y'/'N', default 'Y'), refresh_max_archives_pct (int as string, default '5'), refresh_max_bytes (int as string, default '2000000000') — see "Archive refresh" above. There is no schema-level default: rows simply may not exist for a given config, so refresh code reads these with .get(key, default).

backup_client_configs_exclusions — (client_fqdn, backup_root, pattern), PK all three. pattern is a Python regex matched with fullmatch against /-prefixed paths relative to backup_root (see ClientConfig.is_excluded). Manage via deep-freeze-ctl.py exclude … (or raw SQL).

files — one row per known file per config. file_id PK; identity index on (client_fqdn, backup_root, relative_path). Live: size, modification, status, last_archive_id. Staging: new_size, new_modification, new_status, sweep_mark, force_backup. status/new_status ∈ {present, absent}; sweep_mark ∈ {present, absent}; force_backup ∈ {Y,N}.

s3_archives — one row per uploaded archive. archive_id PK; unique (cloud, region, bucket, archive_file_name). total_size, relevant_size (bytes still current), status ∈ {pending_upload, uploaded, pending_deletion, deleted}, sha256 (unused TODO), created (default current_timestamp, UTC). archive_file_name is stored without the .enc suffix; the S3 object has .enc appended.

file_archive_records — junction of files↔archives. PK (file_id, archive_id), index on archive_id. file_size, file_modification, status ∈ {pending_upload, uploaded, superseded, deleted}. Multiple records per file across time = the backup history; the uploaded one is the current copy.

Encryption

Symmetric only, via gpg -c --pinentry-mode=loopback --passphrase-file <key_file>. The key file is per-config (key_file_path); losing it means losing the backups, so it must be stored safely (password manager). Asymmetric/public-key encryption is a repeatedly-noted TODO.

Scripts

  • deep-freeze.py — run all non-manual backups (the launchd entry point via wrapper deep-freeze).
  • deep-freeze (bash) — guarded wrapper for scheduled runs: skips unless on network, not on a disallowed_ssids (metered) network, and last success ≥ 24h ago; touches ~/.deep-freeze-backups/last_successful_backup on success. Config sourced from ~/.deep-freeze-backups/deep-freeze.rc. Uses freedesktop exit-code conventions.
  • deep-freeze-manual.py --client-name --backup-root [--cloud-provider --region] — runs a single config (used for manual_only ones) via Coordinator.run_manual.
  • create-config.py — writes a backup_client_configs (+ options) row. Flags: --cloud-provider --region --aws-profile --bucket --client-name --backup-root --key-file [--no-cross-devices] [--manual-only] [--temp-directory]. (For exclusions and full CRUD use deep-freeze-ctl.py.)
  • deep-freeze-ctl.py — management CLI over the local DB (never touches S3). Subcommands: config add|list|show|delete|enable|disable|set-option|unset-option, exclude add|remove|list, files search, archives search|refresh-candidates. config add mirrors create-config.py and additionally takes repeatable --exclude REGEX. config disable sets status='inactive' (a new status value — the run/backup code only ever selects 'active', so this cleanly excludes a config from runs without deleting it). config delete removes the config + its options/exclusions but keeps files/file_archive_records history (re-adding the same config re-attaches it); it prompts unless --force. The search subcommands take SQL LIKE patterns (--pattern) and optional filters. archives refresh-candidates lists a config's refresh queue (see "Archive refresh" above) without running a backup. All new SQL lives as methods on Database in db/db.py.
  • deep-freeze-restore.py --client-name --backup-root --target <relative_path> — does not restore; it prints the sequence of commands to run: restore-object (thaw from Glacier), head-object (poll), s3 cp, gpg -d, tar xzf. Note the MacOS UTF-8 NFC/NFD matching hack in utf8_to_query (uses SQL LIKE wildcards because SQLite compares bytes).
  • report.py — prints per-config archive relevance stats + a matplotlib histogram (needs numpy, matplotlib). Recomputes relevant_size first (fix_stats).

Scheduling (macOS launchd)

install/choose_prefix.deep-freeze.plist is a template (fill [...] placeholders): runs the deep-freeze wrapper every 3600s. Symlink into ~/Library/LaunchAgents and launchctl load.

Tests

test/test.sh builds test/Dockerfile (Ubuntu + python3/gnupg/sqlite3/bats) and runs test/run_tests.bats inside the container (bats-support/bats-assert). The AWS CLI is mocked by test/mock/aws (a script that just exit 0s — no real S3). test/initial_setup.sh seeds three configs (deep-freeze dir, a test_root with exclusions and --no-cross-devices, and a manual_only test_root2 with a temp dir) and a file tree, then the bats cases exercise: incremental backup, exclusions, deletions, supersession, manual backup, restore-command generation, 180-day purge (by back-dating s3_archives.created), progressive archive refresh (partial relevance, the byte cap, and refresh→supersession→purge closing the loop in one run), and the deep-freeze-ctl.py management CLI (config CRUD, exclusion management, files/archives search).

Run locally: cd test && ./test.sh (needs Docker). CI: .github/workflows/test.yaml on push/PR to main runs the same. The bats suite must be run inside a container (initial_setup.sh refuses otherwise) because it writes to ~/.deep-freeze-backups and deletes the DB.

Conventions & gotchas

  • No test runner for the Python beyond the bats end-to-end suite; there are no unit tests.
  • Several limits are hardcoded: 500 MB archive size, 180-day purge age, DB path.
  • SQL is inline-stringed in db/db.py; keep new queries there and parameterize (?).
  • Times: file mtimes stored as UTC strings %Y-%m-%d %H:%M:%S %Z; archive created is UTC.
  • Simultaneous/concurrent backups are prevented by RunLock (backup/lock.py), a non-blocking flock taken for the whole Coordinator.run()/run_manual() call; a second overlapping process raises AlreadyRunning rather than racing the first (see architecture section above).
  • Dev DB path is commented out in db/db.py if you need to point at a scratch DB.
  • Python deps are in .venv (direnv .envrc exposes it); core code imports stdlib only.
  • See TODO.md for the maintainer's roadmap (asymmetric encryption, partial-relevance purge, recursive-glob exclusions once Python 3.13's PurePath.match lands, interruption recovery tests).