Skip to content

bench/docs: add S3 benchmark#4369

Open
ness-david-dedu wants to merge 8 commits intoredpanda-data:mainfrom
ness-david-dedu:feature/aws-benchmarking
Open

bench/docs: add S3 benchmark#4369
ness-david-dedu wants to merge 8 commits intoredpanda-data:mainfrom
ness-david-dedu:feature/aws-benchmarking

Conversation

@ness-david-dedu
Copy link
Copy Markdown
Contributor

Results

  • Kafka Connect S3 Sink: up to ~250k msg/s
  • Kafka Connect S3 Source: up to ~73k msg/s
  • Redpanda Connect (single): ~60k msg/s
  • Redpanda Connect (multi): ~73k msg/s
  • Bucket walk (sequential):
    • Small objects: ~560 msg/s
    • Large objects: ~190 msg/s (~195 MB/s)

Key Takeaways

  • S3 is latency-bound without concurrency
  • Batching (flush.size) is the primary driver for write throughput
  • Kafka Connect scales via parallel S3 writers (~4× higher peak)
  • Redpanda Connect is capped by shared output concurrency
  • Results reflect LocalStack limits, not full S3 scalability

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant