Skip to content

feat(tools/e2e): add limits harness two-tier topology composer (#381) - #401

Merged
Minipada merged 3 commits into
jazzyfrom
feature/381-limits-harness-two-tier-topology-compose
Aug 19, 2026
Merged

feat(tools/e2e): add limits harness two-tier topology composer (#381)#401
Minipada merged 3 commits into
jazzyfrom
feature/381-limits-harness-two-tier-topology-compose

Conversation

@Minipada

Copy link
Copy Markdown
Owner

Summary

  • Adds tools/e2e/scripts/run_limits_two_tier.sh, a new sibling E2E scenario that brings up a shared Postgres + RustFS, an aggregating Shipper (standalone Vector instance, no dc_bridge/ROS involved), N real DC stacks, and M synthetic senders (scripts/load_driver.py, Limits harness: synthetic load driver (shipper ingest protocol sender) #378) on a dedicated podman bridge network.
  • Each real stack's own local Shipper has no direct path to the real Postgres — params/e2e_limits_forward_sink.toml (an ADR-0003 custom_config_files passthrough) relays its dc.<tag> route branches to the aggregating Shipper over Vector's native inter-instance relay protocol instead, standing in for Limits harness: measure where the pipeline saturates, on four axes #323's motivating scenario (a robot with no direct route to the cloud). The aggregating Shipper's own postgres sink is the only thing that writes Measurement/synth Records into the real dc_records table.
  • Reuses verify_zero_loss.py unmodified, per the acceptance criteria. Verified end-to-end for real against a locally built dc-e2e image (rebuilt Containerfile.e2e on the existing cached workspace image — no dc_bridge/measurement_server C++ touched by this ticket): ZERO-LOSS VERIFICATION PASSED (25 sources checked, 0 violations) at default settings.
  • Two discoveries recorded (script header + README), matching the acceptance criteria's own "if that assumption turns out false, record it" hedge:
    1. Files bypass the two-tier chain — the Uploader writes File bytes and its own file-metadata Records directly (AWS SDK/libpq), with no Vector hop at all, so there's no way to route them through the aggregator without new DC code.
    2. The aggregator's single postgres sink is a shared bottleneck no single-tier scenario has; two real stacks reliably produced enough at-least-once re-delivery (confirmed genuine repeats, not loss, via direct Postgres queries) to trip verify_zero_loss.py's zero-tolerance memory-timestamp check. DC_E2E_LIMITS_REAL_STACKS stays configurable but defaults to 1 so the scenario passes reliably on its own, per Limits harness: two-tier topology composer (aggregating Shipper stack) #381's "before any ramping logic exists, at a fixed, non-saturating load" scope — finding the actual bottleneck is the ramp controller's job (Limits harness: measure where the pipeline saturates, on four axes #323's next piece).

Test plan

  • prek run --all-files --skip build-doc — all hooks pass
  • shellcheck tools/e2e/scripts/run_limits_two_tier.sh — clean
  • Ran the full scenario end-to-end against a real, locally built dc-e2e image (default settings): ZERO-LOSS VERIFICATION PASSED (25 sources checked, 0 violations)
  • Verified the underlying vector-sink/vector-source relay mechanism standalone against the pinned Vector build before writing any ROS-side config
  • Not wired into ci.yaml (same as run_retention.sh/run_incident.sh/run_degraded.sh/run_load_driver_shipper_test.sh) — left for a future decision, matching those siblings

Closes #381

🤖 Generated with Claude Code

https://claude.ai/code/session_01AgLSYTiwwZcD1hiVePPWNz

Minipada and others added 3 commits August 19, 2026 18:57
Brings up N real DC stacks + M synthetic senders behind an aggregating
Shipper (a standalone Vector instance) to shared Postgres/RustFS, with
each real stack's own local Shipper relaying its Records to the
aggregator over Vector's native inter-instance protocol (an ADR-0003
custom_config_files passthrough) instead of writing directly — standing
in for #323's motivating scenario, a robot with no direct path to the
cloud. Reuses verify_zero_loss.py unmodified; verified end-to-end
against a locally built dc-e2e image.

Two discoveries recorded in the script/README rather than worked
around: Files bypass the two-tier chain entirely (the Uploader writes
directly, no Vector hop), and the aggregator's single postgres sink is
a shared bottleneck that pushed at-least-once re-delivery past
verify_zero_loss.py's zero-tolerance memory-timestamp check under two
concurrent real stacks in this sandbox — real_stacks defaults to 1
(still configurable) so the scenario passes reliably on its own, per
#381's own "before any ramping logic exists" scope.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgLSYTiwwZcD1hiVePPWNz
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgLSYTiwwZcD1hiVePPWNz
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
…arness-two-tier-topology-compose

Signed-off-by: David Bensoussan <d.bensoussan@proton.me>

# Conflicts:
#	progress.txt
#	tools/e2e/README.md
@Minipada
Minipada merged commit ec0e9d8 into jazzy Aug 19, 2026
6 checks passed
@Minipada
Minipada deleted the feature/381-limits-harness-two-tier-topology-compose branch August 19, 2026 20:17
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.09%. Comparing base (2f0fa42) to head (b3393d6).
⚠️ Report is 1 commits behind head on jazzy.

Additional details and impacted files
@@           Coverage Diff           @@
##            jazzy     #401   +/-   ##
=======================================
  Coverage   68.09%   68.09%           
=======================================
  Files         102      102           
  Lines        6320     6320           
=======================================
  Hits         4303     4303           
  Misses       2017     2017           
Flag Coverage Δ
cpp-jazzy 68.09% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant