Keep block files compressed on disk, decompress a block on the page fault that touches it, compress it again on write-out. The engine is better placed for this than a single-file segment backend, because its persistent form is already one file per block and it already owns a fault handler. It does not fit the current architecture, and the reason is worth writing down, together with what a version that supports it would have to change.
Not planned for the current design round. This is v2 material.
Why it does not fit today
The block file is the memory image. map_block_file (include/privateer/vm.hpp:52) maps a committed block MAP_PRIVATE | MAP_FIXED, PROT_READ directly from its file, and the file must be exactly block_size bytes, because a shorter file turns reads past its end into a runtime SIGBUS. Three properties of the engine rest on that mapping:
- The engine never reads a block. Read faults are served by the kernel from the page cache, with no engine involvement.
region_on_fault (src/region.cpp:357) therefore only ever sees write faults, which is why it takes an address and no read/write classification, and does not receive the ucontext.
- The write barrier is a protection change, not a copy.
protect_slot_for_write flips the slot writable and the kernel copies pages out of the page cache of the block file as they are written. That is what makes a barrier fault about 9.6 microseconds at 2 MiB.
- A commit hands pages back. The write-out replaces the private dirty pages with the new file mapping in one
MAP_FIXED call (src/region.cpp:1959), so a clean block costs page cache instead of anonymous memory. This is where the engine's lower resident peak in the serving phase comes from, about 30 percent below a single-file segment backend at 100 million triples.
A compressed file cannot be mapped, so all three stop working at once. Transparent file-system compression keeps every mapping; engine-side compression of blocks that a live region maps does not.
What a version with compression would have to change
- Block file format v2. A container instead of raw bytes: header with codec, uncompressed size and frame size, a frame index, then independently compressed frames of about 64 to 128 KiB, so one fault decompresses one frame instead of a whole block. The block name stays the digest of the uncompressed bytes. That keeps dedup, keeps two codecs interchangeable for the same content, and leaves recipe format v1 untouched, because the codec is a property of the block file and not of the recipe entry. Cost:
block_store::publish can no longer answer a duplicate with a byte compare of file against memory. It has to decompress frame by frame and compare. Putting the digest in the header instead would remove the collision byte compare, and that compare is the reason a 128 bit non-cryptographic digest is safe here, so it would reopen the hash choice.
- Read fault ownership. A compressed slot has to start as anonymous
PROT_NONE, so every first touch traps. The handler then reads and decompresses, which in signal context means mlocked handler text and a decompressor that does not allocate, or the work moves to userfaultfd with a helper thread, which is also what reports read against write properly. The least invasive shape: any fault decompresses the frame and sets PROT_READ, and a write then re-faults into the existing claim path, so the slot state machine stays as it is.
- A resident budget becomes mandatory. Decompressed pages are private anonymous memory. The kernel can swap them but cannot drop them, and two slots holding the same block no longer share them. The engine has to evict clean decompressed pages itself (
MADV_DONTNEED plus PROT_NONE, then decompress again on the next touch). The sweep machinery exists in src/resident.cpp, but the resident budget was measured and dropped as not worth it; with compression it is not optional.
- Unchanged: publication and the atomic link, the dedup fast path itself, snapshots by hard link or
FICLONE, the durability barrier, the recipe. Compressing in the write-out worker before publish is the easy half of the feature.
Where it pays and where it costs
It attacks the right numbers. A store of 11.5 GiB at 100 million triples, about 21 GiB of free space for a five snapshot retention series at that size, and a checkpoint that rewrites gigabytes of blocks when a small update touches nodes spread over the whole store. Write bandwidth and SSD endurance are the real wins, and hypertrie nodes should compress well.
It costs exactly the arms that are at parity today: query p50 of 0.044 to 0.046 ms warm and 0.046 to 0.053 ms cold, and readers holding 85 to 87 thousand operations per second under a live writer. Those numbers exist because reads never enter the engine. A 64 KiB frame is 15 to 20 microseconds with lz4 and about 60 with zstd level 1, against well under a microsecond for a page cache hit. It also gives back part of the lower resident peak, because that win is the engine returning pages to the page cache, which is what compression removes.
So the gate is capacity, not speed. Compression pays when the working set does not fit in RAM, or when disk footprint and write volume are the binding constraint. It does not pay for a store that fits in page cache.
Cheaper options to try first
- Transparent file-system compression. btrfs or zfs with
compression=zstd is compressed on disk, decompressed on fault and compressed on write, done by the kernel, with every mapping intact and no engine change. It composes with block-granular dedup because a block is a whole file. One benchmark arm with compression on against off, same device, answers what compression buys the real workload before any code exists.
- Compress only blocks that no live region maps, meaning retired blocks pinned solely by snapshot recipes. Same fault path, no residency rework, and it hits the largest measured cost, the retention series. This is the offline tool for cold snapshots that the design already parks as possible future work.
Scope
No work planned here for now. If it is picked up, it is its own design step with format v2, read fault ownership and a resident budget, and the measurement in option 1 should come first.
Keep block files compressed on disk, decompress a block on the page fault that touches it, compress it again on write-out. The engine is better placed for this than a single-file segment backend, because its persistent form is already one file per block and it already owns a fault handler. It does not fit the current architecture, and the reason is worth writing down, together with what a version that supports it would have to change.
Not planned for the current design round. This is v2 material.
Why it does not fit today
The block file is the memory image.
map_block_file(include/privateer/vm.hpp:52) maps a committed blockMAP_PRIVATE | MAP_FIXED, PROT_READdirectly from its file, and the file must be exactlyblock_sizebytes, because a shorter file turns reads past its end into a runtime SIGBUS. Three properties of the engine rest on that mapping:region_on_fault(src/region.cpp:357) therefore only ever sees write faults, which is why it takes an address and no read/write classification, and does not receive the ucontext.protect_slot_for_writeflips the slot writable and the kernel copies pages out of the page cache of the block file as they are written. That is what makes a barrier fault about 9.6 microseconds at 2 MiB.MAP_FIXEDcall (src/region.cpp:1959), so a clean block costs page cache instead of anonymous memory. This is where the engine's lower resident peak in the serving phase comes from, about 30 percent below a single-file segment backend at 100 million triples.A compressed file cannot be mapped, so all three stop working at once. Transparent file-system compression keeps every mapping; engine-side compression of blocks that a live region maps does not.
What a version with compression would have to change
block_store::publishcan no longer answer a duplicate with a byte compare of file against memory. It has to decompress frame by frame and compare. Putting the digest in the header instead would remove the collision byte compare, and that compare is the reason a 128 bit non-cryptographic digest is safe here, so it would reopen the hash choice.PROT_NONE, so every first touch traps. The handler then reads and decompresses, which in signal context means mlocked handler text and a decompressor that does not allocate, or the work moves to userfaultfd with a helper thread, which is also what reports read against write properly. The least invasive shape: any fault decompresses the frame and setsPROT_READ, and a write then re-faults into the existing claim path, so the slot state machine stays as it is.MADV_DONTNEEDplusPROT_NONE, then decompress again on the next touch). The sweep machinery exists insrc/resident.cpp, but the resident budget was measured and dropped as not worth it; with compression it is not optional.FICLONE, the durability barrier, the recipe. Compressing in the write-out worker before publish is the easy half of the feature.Where it pays and where it costs
It attacks the right numbers. A store of 11.5 GiB at 100 million triples, about 21 GiB of free space for a five snapshot retention series at that size, and a checkpoint that rewrites gigabytes of blocks when a small update touches nodes spread over the whole store. Write bandwidth and SSD endurance are the real wins, and hypertrie nodes should compress well.
It costs exactly the arms that are at parity today: query p50 of 0.044 to 0.046 ms warm and 0.046 to 0.053 ms cold, and readers holding 85 to 87 thousand operations per second under a live writer. Those numbers exist because reads never enter the engine. A 64 KiB frame is 15 to 20 microseconds with lz4 and about 60 with zstd level 1, against well under a microsecond for a page cache hit. It also gives back part of the lower resident peak, because that win is the engine returning pages to the page cache, which is what compression removes.
So the gate is capacity, not speed. Compression pays when the working set does not fit in RAM, or when disk footprint and write volume are the binding constraint. It does not pay for a store that fits in page cache.
Cheaper options to try first
compression=zstdis compressed on disk, decompressed on fault and compressed on write, done by the kernel, with every mapping intact and no engine change. It composes with block-granular dedup because a block is a whole file. One benchmark arm with compression on against off, same device, answers what compression buys the real workload before any code exists.Scope
No work planned here for now. If it is picked up, it is its own design step with format v2, read fault ownership and a resident budget, and the measurement in option 1 should come first.