feat(persistence): implement F-04 BGREWRITEAOF with delta-buffered atomic swap#43
Merged
Conversation
Capture the approved design for background AOF compaction: a dedicated OS thread snapshots the live keyspace and writes a compact final-state AOF to a temp file, while the main append path buffers in-flight writes into a bounded delta buffer; an atomic rename plus handle swap plus delta replay swaps it in with zero data loss. Triggered only by the new BGREWRITEAOF command (YAGNI auto-rewrite for v0.5). The compact file is the same RESP2 stream replay already consumes, so the on-disk format is unchanged. Refs F-04
…omic swap - Add Command::RewriteAof to protocol parser with arity check - Add FerrumError::AofNotEnabled for AOF-dependent commands - AofWriter: add delta buffer (64 MiB), rewrite_lock, begin/abort/finish_rewrite, rewrite_temp_path, is_rewriting — serialises appends against atomic file swap - KvEngine::rewrite_aof(): spawn dedicated ferrum-aof-rewrite OS thread, never blocks the caller; idempotent when already rewriting - perform_aof_rewrite(): snapshot keyspace → compact final-state AOF (one SET per key + PEXPIREAT) → rename+swap → delta replay - serialise_compact_aof(): RESP2-identical compact format (A) - Server dispatch for BGREWRITEAOF → engine.rewrite_aof() → +OK Unit tests (engine + AofWriter): - rewrite compacts to one SET per live key - rewrite preserves TTL as PEXPIREAT, drops expired keys - writes during rewrite survive restart (delta buffer) - delta overflow aborts rewrite and keeps old AOF - finish_rewrite atomically swaps handle and replays delta Integration tests (TCP end-to-end): - bgrewriteaof produces compact AOF and restores on restart - write during rewrite survives restart - #[ignore] 1M-key scale test for roadmap exit criterion
wait_for_rewrite now only waits for an in-progress rewrite to finish. Callers are responsible for the start-race check. In very fast environments the rewrite may complete before the caller arrives; the downstream file-content assertions serve as the ultimate correctness check.
Replace fixed 200ms sleep after spawning the server thread with a connection-retry loop. In CI the Tokio runtime inside run_listener may take longer to initialise, causing the TCP connect to fail and all subsequent SET commands to get empty replies.
A raw TCP connect leaks an unconsumed connection handler inside the server, starving the Tokio runtime's worker threads. Use a full PING round-trip instead — the handler processes and returns, freeing the worker for the real test traffic.
Replace the complex server-readiness detection with a simpler pattern: the send helper retries when the server returns an empty response. This handles the CI timing where the Tokio runtime accepts the connection but hasn't fully started processing commands yet.
Remove the connection-polling readiness check which leaks handler tasks on low-core CI runners. Replace with a longer fixed sleep (500ms) and increase the send timeout to 5 seconds — the retry loop in send_with_timeout handles any remaining timing jitter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements F-04: AOF REWRITE — the BGREWRITEAOF command that compacts the append-only file.
Changes
Closes F-04