Feature/mysql benchmarking

[ness-david-dedu]

MySQL Benchmarking Results

Benchmarks cover four scenarios across two tools: Redpanda Connect (mysql_cdc input, sql_insert output) and the Confluent
ecosystem (Debezium source connector, Kafka Connect JDBC Sink). All runs on Intel Core i7-10850H @ 2.70GHz, 32 GB RAM, WSL2.

1. Pure MySQL Read (no Kafka) — mysql_cdc with dropped output

Measures the raw read ceiling with no downstream bottleneck (output is dropped immediately).

┌────────────┬───────────────┬───────────────┬───────────────┐
│ GOMAXPROCS │ batch=1,000 │ batch=5,000 │ batch=10,000 │
├────────────┼───────────────┼───────────────┼───────────────┤
│ 1 │ 99,977 msg/s │ 103,433 msg/s │ 104,630 msg/s │
├────────────┼───────────────┼───────────────┼───────────────┤
│ 2 │ 163,592 msg/s │ 173,022 msg/s │ 173,045 msg/s │
├────────────┼───────────────┼───────────────┼───────────────┤
│ 4 │ 187,419 msg/s │ 187,439 msg/s │ 187,462 msg/s │
├────────────┼───────────────┼───────────────┼───────────────┤
│ 8 │ 191,439 msg/s │ 187,464 msg/s │ 187,464 msg/s │
└────────────┴───────────────┴───────────────┴───────────────┘

Peak: 191K msg/sec / 115 MB/sec. Strong scaling 1→2 cores (~1.67×), plateaus at 4+ cores. Batch size has negligible effect.

2. MySQL → Kafka Snapshot — RPCN vs Debezium

Reading a 10M-row snapshot into a Kafka topic.

┌──────────────────┬──────────────────────────────┬─────────┐
│ Tool │ Config │ msg/sec │
├──────────────────┼──────────────────────────────┼─────────┤
│ Redpanda Connect │ unbounded cores, batch=10000 │ 50,440 │
├──────────────────┼──────────────────────────────┼─────────┤
│ Debezium Source │ batch=5000, queue=20000 │ 13,386 │
└──────────────────┴──────────────────────────────┴─────────┘

RPCN is ~4× faster than Debezium for snapshot reads. Batch size matters most at 1 core (15K→46K); beyond batch=5000 at 2+
cores, the bottleneck shifts to Kafka.

3. Kafka → MySQL Write — RPCN vs Kafka Connect JDBC Sink

Writing 10M rows from a Kafka topic into MySQL.

┌─────────────────────┬───────────────────────┬─────────┐
│ Tool │ Config │ msg/sec │
├─────────────────────┼───────────────────────┼─────────┤
│ Redpanda Connect │ 4 cores, batch=10000 │ 64,102 │
├─────────────────────┼───────────────────────┼─────────┤
│ JDBC Sink connector │ 16 tasks, batch=10000 │ 43,859 │
└─────────────────────┴───────────────────────┴─────────┘

RPCN is ~1.5× faster using significantly fewer resources (4 cores vs 16 connector tasks).

4. CDC Streaming — RPCN vs Debezium

Streaming 10M live insert events into Kafka.

┌──────────────────┬─────────────────────────┬─────────┐
│ Tool │ Config │ msg/sec │
├──────────────────┼─────────────────────────┼─────────┤
│ Debezium Source │ batch=5000, queue=20000 │ 25,510 │
├──────────────────┼─────────────────────────┼─────────┤
│ Redpanda Connect │ 1 core, batch=5000 │ 19,920 │
└──────────────────┴─────────────────────────┴─────────┘

The only benchmark where Debezium wins (~1.3× faster). RPCN CDC throughput is flat across 1→8 cores