docs(adr): decide pipeline architecture v2 (adopt as target, gate graduation)#2522
Merged
Conversation
…aduation Resolve the open question of what to do with the v2 pipeline architecture (pkg/lifecycle-poc), which shipped as an undocumented, incomplete, unbenchmarked preview that taxes every data-path change. Benchmarked both architectures on the same 100k-record workload (committed: BenchmarkStreamOld vs the existing BenchmarkStreamNew). v2 allocates ~6.3× fewer objects and ~3.3× less memory per record — a real, batching-driven data-path efficiency win. So: adopt v2 as the target, keep it behind the preview flag with v1 default until explicit graduation criteria are met (multi-destination, recovery parity, chaos-tested correctness, an end-to-end benchi confirmation, Tier-1 sign-off), and do not delete it. Documents the decision and its finish line so the fork stops being a silent tax on future PRs. Adds the v1 counterpart benchmark for reproducibility of the claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves the open question of what to do with the v2 pipeline architecture (
pkg/lifecycle-poc) — an undocumented, incomplete (single source/dest), unbenchmarked preview that taxes every data-path change (it had to be reasoned about for #2268, SIGTERM, and the #2519 force-stop follow-up).Decision — driven by a benchmark, not a hunch
Benchmarked both architectures on the same 100k-record workload (committed, reproducible; no-op connectors isolating data-path orchestration):
The batching amortizes per-record channel/ack overhead that v1 pays across four nodes on every record. Allocation/GC pressure drives throughput and tail latency under load, so this is a real efficiency win.
So: adopt v2 as the target architecture; keep it behind
-preview.pipeline-arch-v2(v1 default) until explicit graduation criteria are met — multi-destination, recovery parity, a chaos-test correctness bar (invariants 1–7 under SIGKILL mid-batch), an end-to-end benchi confirmation, and Tier-1 sign-off (realistically Phase 2 + a second maintainer). Do not delete it (win is proven); do not make it default prematurely (it's incomplete/unrecovered).The ADR also bounds the interim dual-maintenance tax (portability rule + explicit gap-tracking) and includes a supersession clause if a future benchi run shows no end-to-end win.
Files
docs/architecture-decision-records/20260704-pipeline-architecture-v2.mdpkg/lifecycle/stream/benchmark_test.go— the v1 counterpart benchmark (reproducibility of the claim)Tier 3 (docs + a test-only benchmark). Advances the "no undocumented architectural forks" discipline.
🤖 Generated with Claude Code