Per-push batch-frame compression + transport file split#7
Merged
Conversation
The transport package was one ~820-line file. Split it along the three jobs it already did, no logic change: - export.go: encode checkpoints to the wire format and commit them to the orphan branch (ExportNewFrames/ExportAllFrames/CommitWireFormat/ EnsureOrphanBranch). - import.go: decode a branch back into data.db (ImportBranch). - sync.go: fetch teammate branches and import them into the index (FetchRemoteRekalRefs/ListRemoteRekalBranches/ImportBranchToIndex). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBMA7dNfAQn86m59P64dsS
The wire format compressed each frame as its own zstd stream, so a frame could never back-reference the paths, emails, ULIDs, or phrasing repeated in its sibling frames, and paid zstd framing overhead once per frame. On small frames that is the dominant cost — the preset dictionary was largely compensating for the isolation. Add a batch frame (type 0x06): one checkpoint's session frames plus its checkpoint frame are concatenated uncompressed (each member prefixed with type + varint length) and compressed together as a single stream. Members carry the exact bytes a standalone frame would, so they decode with the existing per-type parsers after one decompress. On a redundant multi-session push this is multiple-x smaller (6.8x in the codec test's synthetic case; realistically ~2-4x). Backwards compatibility is the same contract the v2 frames use, and is tested in all four directions: - new reads new: batch round-trip + the E2E push/clone/import test. - new reads old: existing standalone-frame tests still pass; import and sync keep handling 0x01-0x05 frames, and bodies may mix old frames with new batches (TestBatch_MixedBody). - old reads new: a pre-batch binary skips a 0x06 frame by its envelope length and still reads the frames around it (TestBatch_OldReaderSkips). - corrupt/truncated batches error, never panic (truncation sweep + oversized-member-length guard). Export batches per checkpoint, falling back to standalone frames if a single checkpoint would overflow the envelope's u24 length. Import and sync-to-index decode batches by dispatching each member through the same handlers as standalone frames. Format documented in docs/git-transportation.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBMA7dNfAQn86m59P64dsS
TestPush_E2E_ExportAndPush asserted the pushed body's raw frame layout, which the batch frame deliberately changed: a single-checkpoint push is now one batch frame (session + checkpoint grouped) plus a meta frame — 2 frames, not 3; two pushes give 4, not 6. Update the counts and, instead of decoding standalone session/checkpoint frames, decode the batch and assert on its members — which also exercises DecodeBatch end-to-end. The append-only prefix check is unchanged and confirms batching preserves that property. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TBMA7dNfAQn86m59P64dsS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes on top of the earlier refactors. CI green (incl. the E2E push→clone→import roundtrip through DuckDB and the team-sync import path).
Per-push batch-frame compression (the substance)
The wire format compressed each frame as its own zstd stream, so a frame could never back-reference the paths, emails, ULIDs, or phrasing repeated in its sibling frames, and paid framing overhead per frame — the dominant cost on small frames.
New batch frame (
0x06): one checkpoint's session frames plus its checkpoint frame are concatenated uncompressed (each member =type + varint length + payload) and compressed together as a single stream. Members carry the exact bytes a standalone frame would, so they decode with the existing per-type parsers after one decompress. On a redundant multi-session push this is multiple-x smaller (6.8× in the codec test's synthetic case; realistically ~2–4×).Backwards compatibility — the same contract the v2 frames use, tested in all four directions:
0x01–0x05handlers, and bodies may interleave old frames with new batches.0x06frame by its envelope length and still reads the frames around it.Export batches per checkpoint (falling back to standalone frames if a single checkpoint would overflow the envelope's u24 length). Import and sync-to-index decode batches by dispatching each member through the same handlers as standalone frames. No data migration; existing branches stay readable. Documented in
docs/git-transportation.md.transport.go file split
The
transportpackage's single ~820-line file split by concern intoexport.go/import.go/sync.go. No logic change.🤖 Generated with Claude Code
https://claude.ai/code/session_01TBMA7dNfAQn86m59P64dsS
Generated by Claude Code