feat(dc_bridge): extract the Uploader into its own dc_uploader process - #459
Merged
Minipada merged 1 commit intoAug 31, 2026
Merged
Conversation
Closes #446 The Uploader used to run on a worker thread inside dc_bridge, configured by ROS parameters and linking the AWS SDK into the Bridge's own process. It is now dc_uploader, a standalone executable configured entirely by DC_UPLOADER_* environment variables, with no ROS dependency. The Bridge keeps only the Files subscription and the durable intent-queue write side; dc_uploader reads that queue, uploads, and emits the resulting status Records itself. IntentQueue gains rescan() so a reader-only instance in a separate process can discover intents a different process enqueued — the queue's on-disk format was already crash-atomic and safe for this, but each process's in-memory scheduling state previously only ever loaded what was on disk at construction time. dc_bringup.launch.py starts dc_uploader automatically alongside dc_bridge (same ExecuteProcess pattern as dc_mcap_writer) whenever a `receives: files` Destination is configured, translating that Destination plus files.*/uploader.data_dir into dc_uploader's environment — deployments keep the same params file. See docs/adr/0014-uploader-runs-as-its-own-process.md for the full decision. Verified with a full tools/e2e/scripts/build.sh run (colcon build + test): 656 tests, 0 failures, and a live smoke test of dc_uploader against RustFS. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VXhyy2nX5JokxtTnGdwzjy Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## jazzy #459 +/- ##
==========================================
+ Coverage 71.00% 71.43% +0.43%
==========================================
Files 120 123 +3
Lines 7462 7553 +91
==========================================
+ Hits 5298 5395 +97
+ Misses 2164 2158 -6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-actions
Bot
deleted the
feature/446-extract-the-uploader-into-its-own-proces
branch
August 31, 2026 16:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #446
dc_bridge. It is nowdc_uploader, a standalone executable in thedc_bridgecolcon package, configured entirely byDC_UPLOADER_*environment variables with no ROS dependency — norclcpp, no ROS params.dc_bridgekeeps only the Files subscription and the durable intent-queue write side (ADR-0005/DC 2.0: durable Uploader intent queue — File uploads survive Bridge restarts #265); it no longer links the AWS SDK at all.IntentQueuegainsrescan(): the queue's on-disk format was already crash-atomic and multi-process-safe, but each process's in-memory scheduling state previously only ever loaded from disk once, at construction. A separatedc_uploaderprocess holding its ownIntentQueueinstance over the same directory needed a way to learn about intents the Bridge process enqueues after that —rescan()is that mechanism, polled every cycle.dc_bringup.launch.pystartsdc_uploaderautomatically alongsidedc_bridge(sameExecuteProcesspattern already used fordc_mcap_writer) whenever areceives: filesDestination is configured, translating that Destination plusfiles.*/uploader.data_dirintodc_uploader's environment — so every existing params file (tools/e2e/params/*.yaml, the demo params, etc.) keeps working unchanged.docs/adr/0014-uploader-runs-as-its-own-process.md.Acceptance criteria from #446:
IntentQueuecrash-replay guarantee, exercised per-process viarescan())dc_uploader)dc_uploadernow runs inside the same container alongsidedc_bridge, both launched by the existingdc_bringup.launch.py)Still one machine at this point (per the issue) — containerizing
dc_uploaderinto its own image/container is #447's work, not this change's.Test plan
tools/e2e/scripts/build.shrun:colcon build+colcon test— 656 tests, 0 errors, 0 failures, 0 skipped.prek run --all-files --skip build-doc— all hooks pass (clang-format, ruff, REUSE/SPDX, etc.).dc_uploaderin the built container against live RustFS withDC_UPLOADER_*env vars set — started, connected, and shut down cleanly on SIGTERM.dc_uploaderwith no env vars — fails fast with a clear "required environment variable ... is not set" message instead of a crash.process_config_test.cpp(env-var parsing/validation) and newIntentQueuerescan tests inintent_queue_test.cpp.🤖 Generated with Claude Code
https://claude.ai/code/session_01VXhyy2nX5JokxtTnGdwzjy