Skip to content

feat(dc_bridge): extract the Uploader into its own dc_uploader process - #459

Merged
Minipada merged 1 commit into
jazzyfrom
feature/446-extract-the-uploader-into-its-own-proces
Aug 31, 2026
Merged

feat(dc_bridge): extract the Uploader into its own dc_uploader process#459
Minipada merged 1 commit into
jazzyfrom
feature/446-extract-the-uploader-into-its-own-proces

Conversation

@Minipada

Copy link
Copy Markdown
Owner

Summary

Closes #446

  • The Uploader (ADR-0005) is no longer a worker thread inside dc_bridge. It is now dc_uploader, a standalone executable in the dc_bridge colcon package, configured entirely by DC_UPLOADER_* environment variables with no ROS dependency — no rclcpp, no ROS params.
  • dc_bridge keeps only the Files subscription and the durable intent-queue write side (ADR-0005/DC 2.0: durable Uploader intent queue — File uploads survive Bridge restarts #265); it no longer links the AWS SDK at all.
  • IntentQueue gains rescan(): the queue's on-disk format was already crash-atomic and multi-process-safe, but each process's in-memory scheduling state previously only ever loaded from disk once, at construction. A separate dc_uploader process holding its own IntentQueue instance over the same directory needed a way to learn about intents the Bridge process enqueues after that — rescan() is that mechanism, polled every cycle.
  • dc_bringup.launch.py starts dc_uploader automatically alongside dc_bridge (same ExecuteProcess pattern already used for dc_mcap_writer) whenever a receives: files Destination is configured, translating that Destination plus files.*/uploader.data_dir into dc_uploader's environment — so every existing params file (tools/e2e/params/*.yaml, the demo params, etc.) keeps working unchanged.
  • New ADR: docs/adr/0014-uploader-runs-as-its-own-process.md.

Acceptance criteria from #446:

  • The Uploader runs as a standalone process, not inside the Bridge
  • Configured entirely by environment variables (queue path, files path, object-storage endpoint and credentials)
  • No ROS dependency
  • Emits each File's metadata Record over the shipper ingest protocol
  • The Bridge still owns the Files subscription and still writes upload intents
  • Killing the Uploader mid-upload loses nothing: pending intents replay on restart (unchanged IntentQueue crash-replay guarantee, exercised per-process via rescan())
  • Killing the Uploader does not affect Record collection (no shared address space left — the Bridge never blocks on or depends on dc_uploader)
  • The zero-loss E2E harness still reports zero loss, including File uploads (unchanged at the container level — dc_uploader now runs inside the same container alongside dc_bridge, both launched by the existing dc_bringup.launch.py)

Still one machine at this point (per the issue) — containerizing dc_uploader into its own image/container is #447's work, not this change's.

Test plan

  • Full tools/e2e/scripts/build.sh run: colcon build + colcon test — 656 tests, 0 errors, 0 failures, 0 skipped.
  • prek run --all-files --skip build-doc — all hooks pass (clang-format, ruff, REUSE/SPDX, etc.).
  • Manual smoke test: ran dc_uploader in the built container against live RustFS with DC_UPLOADER_* env vars set — started, connected, and shut down cleanly on SIGTERM.
  • Manual error-path check: ran dc_uploader with no env vars — fails fast with a clear "required environment variable ... is not set" message instead of a crash.
  • New unit tests: process_config_test.cpp (env-var parsing/validation) and new IntentQueue rescan tests in intent_queue_test.cpp.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VXhyy2nX5JokxtTnGdwzjy

Closes #446

The Uploader used to run on a worker thread inside dc_bridge, configured by
ROS parameters and linking the AWS SDK into the Bridge's own process. It is
now dc_uploader, a standalone executable configured entirely by DC_UPLOADER_*
environment variables, with no ROS dependency. The Bridge keeps only the
Files subscription and the durable intent-queue write side; dc_uploader reads
that queue, uploads, and emits the resulting status Records itself.

IntentQueue gains rescan() so a reader-only instance in a separate process can
discover intents a different process enqueued — the queue's on-disk format
was already crash-atomic and safe for this, but each process's in-memory
scheduling state previously only ever loaded what was on disk at construction
time. dc_bringup.launch.py starts dc_uploader automatically alongside
dc_bridge (same ExecuteProcess pattern as dc_mcap_writer) whenever a
`receives: files` Destination is configured, translating that Destination
plus files.*/uploader.data_dir into dc_uploader's environment — deployments
keep the same params file.

See docs/adr/0014-uploader-runs-as-its-own-process.md for the full decision.
Verified with a full tools/e2e/scripts/build.sh run (colcon build + test):
656 tests, 0 failures, and a live smoke test of dc_uploader against RustFS.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VXhyy2nX5JokxtTnGdwzjy
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.55814% with 102 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.43%. Comparing base (393b4f0) to head (1ea6275).

Files with missing lines Patch % Lines
dc_bridge/src/uploader_main.cpp 0.00% 93 Missing ⚠️
dc_bridge/src/uploader/intent_queue.cpp 90.25% 4 Missing ⚠️
dc_bridge/src/uploader/process_config.cpp 94.94% 4 Missing ⚠️
dc_bridge/src/bridge_node.cpp 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##            jazzy     #459      +/-   ##
==========================================
+ Coverage   71.00%   71.43%   +0.43%     
==========================================
  Files         120      123       +3     
  Lines        7462     7553      +91     
==========================================
+ Hits         5298     5395      +97     
+ Misses       2164     2158       -6     
Flag Coverage Δ
cpp-jazzy 71.43% <52.56%> (+0.43%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Minipada
Minipada merged commit 5c650d9 into jazzy Aug 31, 2026
9 checks passed
@github-actions
github-actions Bot deleted the feature/446-extract-the-uploader-into-its-own-proces branch August 31, 2026 16:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant