Add ARC support #14144
Add ARC support #14144
Conversation
raghu999
requested review from
a team,
bogdandrutu,
dmathieu,
dmitryax and
mx-psi
as code owners
|
|
Added the test results and generated metrics |
| } | ||
| } | ||
|
|
||
| // Acquire obtains a permit unless ARC is disabled or the context is cancelled. |
There was a problem hiding this comment.
My main feedback is this: we have here a large piece of code, it looks good, but it is very specific and opinionated. I would prefer to see this become an extension implementation. This PR is great, I just think it belongs in //extensions.
See #13902 explaining how extension APIs are added, it would begin with //extensions/extensioncontroller APIs, for example, then the bulk of the code would land in //extensions/arccontrollerextension (could be in contrib).
Could we invent an extension point to let you insert your ARC controller into the send pipeline for all exporters? Then, exporterhelper would only need to load and insert the extensions into the pipeline.
There was a problem hiding this comment.
Hi @jmacd, thanks for the detailed feedback on moving this to an extension.
My original thinking was that putting ARC in exporterhelper would make it easier to enable by default for any exporter that is using sending queue after ARC is battle-tested.
This isn't just a theoretical feature for us; it's based on our own painful experience of overwhelming and taking down downstream systems. We operate at a large scale, using both Vector and OTel to ingest hundreds of TBs of telemetry data, and we've seen the critical need for this kind of adaptive backpressure.
This experience is what led us to look for proven solutions. The Netflix tech blog (https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581) inspired Vector's implementation (https://vector.dev/blog/adaptive-request-concurrency/). We've seen the wins from this model firsthand in our own Vector deployments, and it's a key reason they enable it by default. My goal is to bring that same proven stability to OTel.
That said, I'm completely fine with moving this to an extension if the team prefers that. I did talk with @atoulme at KubeCon about how the sending_queue manages its goroutine pool, and I believe this implementation fits in correctly.
Since it's a big architectural choice, I will wait for feedback from the other exporterhelper code owners. @bogdandrutu and @dmitryax, I'd strongly request you read the two blog posts above if you have a moment. They provide the full context for why this feature is so important and why I'm proposing it.
I'll hold off on the refactoring until you've all had a chance to weigh in. Please let me know which direction you'd like me to take. I'm happy to go whichever way the project prefers.
There was a problem hiding this comment.
Here are the gradient-based ARC implementations from Netflix and Envoy.
There was a problem hiding this comment.
Hi @raghu999. We are currently focused on Collector stabilization, which includes relying on the configuration already provided by exporterhelper. The change here adds a significant amount of new logic with its own configuration interface, and even though the intention is great, it does conflict with the stabilization effort.
Did you have a chance to build the Collector with this change and verify that it solves the problems you’re running into? Having more details on how this helps with the challenges from actual experiments would help us figure out if and how this functionality could be incorporated into the Collector.
There was a problem hiding this comment.
Hi @dmitryax, totally understand and appreciate the focus on stabilization. My intention with this PR is to support that effort: today the sending_queue can become a stability bottleneck when static concurrency limits are misconfigured. ARC gives the collector a built-in safety valve that adapts to downstream capacity and prevents overload conditions.
To your question: yes, I have already built and verified the implementation.
So far, we’ve tested this with otlpgrpc at a smaller scale to confirm that configuration parameters are picked up correctly and the control logic activates as expected.
For validation:
- Production: We are currently in a holiday code freeze. Immediately after the Thanksgiving moratorium, I will deploy this to a couple of our production tenants and run it against multiple real downstream exporters. I will share the runtime metrics and results as soon as they’re available.
- Benchmarks: I am currently adding a benchmark suite (BenchmarkARC_vs_Static) to simulate "flapping" downstream destinations (latency spikes and 429s). This will allow us to reproducibly demonstrate that ARC improves stability compared to static limits under controlled load.
A few notes on code impact:
- Strict opt-in: All logic is gated behind qCfg.Arc.Enabled (defaults to false). In NewQueueSender, if ARC is disabled, we return the standard QueueBatch struct immediately. The ARC controller initialization and the export function wrapper are completely skipped, ensuring zero runtime overhead or code path changes for existing users.
- Config isolation: The new arc configuration block is fully self-contained and does not alter any existing queue/batch semantics.
- Feature Gate option: As suggested by @atoulme during KubeCon, I’m fully open to putting the entire ARC functionality behind an Alpha Feature Gate (e.g., exporter.adaptiveRequestConcurrency). If the team feels that would be the safest path during the stabilization period, I can add that in this PR.
I’ll follow up with the benchmark data shortly. Happy to iterate on the Feature Gate approach if that helps merge this safely during the freeze!
There was a problem hiding this comment.
The benchmark suite (queue_sender_benchmark_test.go) validates ARC across 6 key objectives:
- Baselines: Verifies negligible overhead by comparing ARC (enabled/disabled) against the current static implementation in ideal conditions.
- Latency Compensation: Tests ARC's ability to scale concurrency (up to 100) to maintain throughput when downstream latency increases (10ms).
- Backpressure & Safety: Simulates a 50% error rate (e.g., HTTP 429) to verify ARC shrinks concurrency to relieve pressure, compared to static limits which cause contention.
- Jitter Stability: Ensures the algorithm remains stable under noisy network conditions (latency fluctuating 5-15ms).
- Stress Testing: Pushes high concurrency (200) to check for CPU/locking overhead.
- Recovery: Validates fast recovery when latency drops suddenly (100ms → 0ms) or when high static concurrency meets error spikes ("Worst Case").
go test -bench=BenchmarkQueueSender_ARC -benchmem -cpu 1 ./...
PASS
ok go.opentelemetry.io/collector/exporter/exporterhelper 0.368s
goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/collector/exporter/exporterhelper/internal
cpu: Apple M2 Pro
BenchmarkQueueSender_ARC/Baseline_Static_NoLatency 598387 1703 ns/op 1421 B/op 9 allocs/op
BenchmarkQueueSender_ARC/Baseline_ARC_Disabled_NoLatency 1000000 1489 ns/op 1392 B/op 9 allocs/op
BenchmarkQueueSender_ARC/Baseline_ARC_Enabled_Steady 1000000 1398 ns/op 1330 B/op 8 allocs/op
BenchmarkQueueSender_ARC/Stress_ARC_200Concurrent_NoLatency 939532 1215 ns/op 1435 B/op 8 allocs/op
BenchmarkQueueSender_ARC/HighLatency_Static_10Conns 1090 1092275 ns/op 1396 B/op 11 allocs/op
BenchmarkQueueSender_ARC/HighLatency_ARC_10to100Conns 1000000 1214 ns/op 1343 B/op 8 allocs/op
BenchmarkQueueSender_ARC/Jittery_ARC_5-15ms 1000000 1225 ns/op 1386 B/op 8 allocs/op
BenchmarkQueueSender_ARC/Backpressure_Static_ErrorSpike 10000 115192 ns/op 1489 B/op 12 allocs/op
BenchmarkQueueSender_ARC/Backpressure_ARC_ErrorSpike 996349 1205 ns/op 1174 B/op 8 allocs/op
BenchmarkQueueSender_ARC/Recovery_ARC_100ms_to_0ms 1066398 1223 ns/op 1173 B/op 8 allocs/op
BenchmarkQueueSender_ARC/WorstCase_Static_100Conns_ErrorSpike 95211 12687 ns/op 1469 B/op 12 allocs/op
BenchmarkQueueSender_ARC/WorstCase_ARC_100Conns_ErrorSpike 1000000 1329 ns/op 1505 B/op 8 allocs/op
PASS
There was a problem hiding this comment.
@raghu999 to move this PR forward, I encourage you to explore extracting an extension interface.
I agree that this is important, and I think others do too, but that does not mean that the ARC logic absolutely needs to be interwoven with the rest of exporterhelper.
In terms of an extension interface, I think what we probably want is a higher level version of HTTP client middleware. Instead of operating in terms of HTTP requests/responses, it would operate in terms of the exporterhelper Request interface.
There was a problem hiding this comment.
@axw Thanks for the feedback. I understand the preference for keeping exporterhelper lean and avoiding opinionated logic in the core.
Why I originally proposed exporterhelper: My long-term goal was that once ARC is battle-tested, we could enable it by default. This would allow every exporter that uses sending_queue (Elastic, OTLP, Splunk, etc.) to inherit this stability "for free" without users needing to manually configure an extension for each one.
However, I agree that starting as an extension is the safer architectural path to prove that stability first.
Performance Baseline (Production Elastic): To address @dmitryax's request for verification against real-world challenges, I documented the performance of the current implementation against a Production Elastic backend to capture real-world network/TCP dynamics. I am attaching report.zip, which contains the full breakdown, benchstat analysis, and stress test results.
(Note: I intended to validate against Datadog and Splunk HEC as well, but due to holiday moratoriums I only tested with a real Elastic backend. However, the dynamics—TCP backpressure and variable latency—are universal.)
The results confirm why this logic is critical:
-
5x Throughput: ARC sustained 20k/s vs Static's 4k/s in high-latency scenarios.
-
Better Memory Profile: ARC kept the queue size ~35% smaller (p95) while processing significantly more volume.
-
Non-Blocking: In the benchmarks, ARC reduced scheduling overhead from ~1ms (static lock contention) to ~150ns under load.
Plan: I will proceed with the suggested architecture:
-
Define a generic Controller interface in exporterhelper.
-
Move the AIMD/EWMA logic to a new extension.
-
Wire them up via config.
I will ping you once the interface is ready for review.
There was a problem hiding this comment.
@axw @jmacd @dmitryax I have opened a new PR (#14318) for the ARC controller interface. Should we close this PR in favor of the new one?
Also, the implementation for the ARC extension is complete and will be stacked on top of #14318. Please let me know if I should submit that PR now or wait for the interface to be reviewed first.
There was a problem hiding this comment.
I think we can go ahead and close this one. It'll still be here for reference.
I recommend we get the interface settled before going ahead with the ARC extension implementation.
raghu999
force-pushed
the
arc-feature
branch
from
7a98d38 to
b1c96fb
Compare
|
This was brought up at KubeCon. I think we need to discuss this at a SIG meeting. |
CodSpeed Performance ReportMerging #14144 will improve performances by 35.05%Comparing
|
| Benchmark | BASE |
HEAD |
Change | |
|---|---|---|---|---|
| ⚡ | BenchmarkSplittingBasedOnItemCountHugeLogs |
46.7 ms | 34.6 ms | +35.05% |
5 participants
Description
This PR introduces an Adaptive Request Concurrency (ARC) controller to the
exporterhelper.When enabled via the new
sending_queue.arc.enabledflag, this controller dynamically manages the number of concurrent export requests, effectively overriding the staticnum_consumerssetting. It adjusts the concurrency limit based on observed RTT (Round-Trip Time) and backpressure signals (e.g., HTTP 429/503, gRPCResourceExhausted/Unavailable).The controller follows an AIMD (Additive Increase, Multiplicative Decrease) pattern to find the optimal concurrency limit, maximizing throughput during healthy operation and automatically backing off upon detecting export failures or RTT spikes.
This feature is disabled by default and introduces no behavior change unless explicitly enabled. It also adds a new set of
otelcol_exporter_arc_*metrics (detailed in the documentation) for observing its behavior.Link to tracking issue
Fixes #14080
Testing
internal/arc/controller_test.go, covering additive increase, multiplicative decrease (TestAdjustIncreaseAndDecrease), and the cold-start backoff heuristic (TestEarlyBackoffOnColdStart).shrinkSem(a custom shrinkable semaphore) to validate its concurrency, prioritization, and shutdown safety.TestController_Shutdown_UnblocksWaiters) to ensure that any goroutines blocked onAcquireare correctly unblocked with a shutdown error, preventing collector hangs.internal/queue_sender_test.go(TestQueueSender_ArcAcquireWaitMetric) that validates the end-to-end flow. It confirms that when the limit is reached, new requests block onAcquireand theexporter_arc_acquire_wait_msmetric records the wait time.internal/experr/back_pressure.goutility to verify its detection logic.Documentation
exporterhelper/README.mdto include the newsending_queue.arcblock with all its configuration options.exporterhelper/metadata.yamlto define all newotelcol_exporter_arc_*metrics, which are in turn reflected in the generateddocumentation.md.