Committed benchmarks in conduit-connector-protocol (pconnector/v2/record_marshal_bench_test.go, added in ConduitIO/conduit-connector-protocol#260) show that structured record payloads cost ~5–7× raw payloads on the per-record marshal/unmarshal hot path — the hottest path in the whole system (every record crosses it, twice: source→engine and engine→destination).
Numbers (Apple M3 Max, single-record batches, -benchtime 2s)
| Path |
raw |
structured |
ratio |
| Marshal (record → proto → bytes) |
~1.3 µs, 29 allocs |
~7.8 µs, 125 allocs |
~5.5× |
| Unmarshal (bytes → proto → record) |
~1.5 µs, 46 allocs |
~8.5 µs, 201 allocs |
~5× |
Where the cost is
The overhead is isolated to the opencdc record conversion in this repo — Record.ToProto/FromProto and Change.ToProto/FromProto — specifically the map[string]any ↔ structpb.Struct conversion for structured payloads (opencdc.StructuredData). structpb allocates a wrapper *structpb.Value per field (and recursively per nested field/element), which is why nested structured records dominate.
Direction
- Profile
ToProto/FromProto for structured payloads and reduce per-field allocations (candidates: a custom, allocation-lean map[string]any ↔ proto encoding instead of structpb; pooling; or a flatter wire representation for structured data).
- Any change here is on a serialized/protocol-adjacent path — must preserve exact round-trip semantics (the protocol repo's round-trip tests + the new benchmarks are the guard). No wire-format break without a versioning discussion.
- Re-run the protocol benchmarks before/after to quantify the win (the committed benchmarks make this a clean regression/benefit measurement).
Not urgent, but it's the single highest-leverage micro-optimization in the data path — it benefits every structured-record pipeline.
Committed benchmarks in
conduit-connector-protocol(pconnector/v2/record_marshal_bench_test.go, added in ConduitIO/conduit-connector-protocol#260) show that structured record payloads cost ~5–7× raw payloads on the per-record marshal/unmarshal hot path — the hottest path in the whole system (every record crosses it, twice: source→engine and engine→destination).Numbers (Apple M3 Max, single-record batches,
-benchtime 2s)Where the cost is
The overhead is isolated to the
opencdcrecord conversion in this repo —Record.ToProto/FromProtoandChange.ToProto/FromProto— specifically themap[string]any↔structpb.Structconversion for structured payloads (opencdc.StructuredData).structpballocates a wrapper*structpb.Valueper field (and recursively per nested field/element), which is why nested structured records dominate.Direction
ToProto/FromProtofor structured payloads and reduce per-field allocations (candidates: a custom, allocation-leanmap[string]any↔ proto encoding instead ofstructpb; pooling; or a flatter wire representation for structured data).Not urgent, but it's the single highest-leverage micro-optimization in the data path — it benefits every structured-record pipeline.