feat(dc_bridge): split shipper.data_dir into separate Shipper and Uploader directories - #453
Merged
Minipada merged 3 commits intoAug 31, 2026
Conversation
…oader directories shipper.data_dir fed both the Shipper's disk buffer and the Uploader's durable intent queue / multipart-resume state, tying them to one directory for no architectural reason. Add uploader.data_dir, defaulting to shipper.data_dir so existing deployments keep working unchanged, so each can later be mounted as its own volume. Closes #441 Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
…_dir example Same-path examples obscured which files on disk belong to which owner. Show separate paths in the files-destination example so it's obvious by default, while the code's uploader.data_dir still falls back to shipper.data_dir when unset, per #441. Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
…harness The zero-loss harness only ever ran with uploader.data_dir unset, so it only proved the same-directory default still worked, not that the split from #441 actually functions. Point uploader.data_dir at its own directory in e2e_params.yaml, mounted on a new dc_e2e_uploader volume (separate from the Shipper's dc_e2e_buffer) so it survives the induced outage + restart like the Shipper's buffer already does, and update the durable-intent-queue check to read from the right volume. Also fixes a stale README reference to the queue living under shipper.data_dir. Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## jazzy #453 +/- ##
==========================================
- Coverage 70.93% 70.92% -0.00%
==========================================
Files 119 119
Lines 7401 7402 +1
==========================================
Hits 5249 5249
- Misses 2152 2153 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-actions
Bot
deleted the
feature/441-split-shipper-data-dir-into-separate-shi
branch
August 31, 2026 11:07
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
shipper.data_dirfed both the Shipper's disk buffer (rendered into Vector's config) and the Uploader's durable intent queue + multipart-resume state — one parameter tied two independent owners to the same directory for no architectural reason.uploader.data_dir, used for<uploader.data_dir>/uploader(multipart-resume state) and<uploader.data_dir>/queue/upload(the durable intent queue, DC 2.0: durable Uploader intent queue — File uploads survive Bridge restarts #265).shipper.data_dir, so a deployment that only ever setshipper.data_dirkeeps working unchanged; the two may still point at the same directory, or be split so each can be mounted as its own volume.Test plan
colcon test(CI)uploader.data_dirdefaults to the existingshipper.data_dirvalue in all e2e param filesCloses #441