Skip to content

perf: reduce allocations in structured-payload record conversion (map[string]any ↔ structpb.Struct) #269

Description

@devarismeroxa

Committed benchmarks in conduit-connector-protocol (pconnector/v2/record_marshal_bench_test.go, added in ConduitIO/conduit-connector-protocol#260) show that structured record payloads cost ~5–7× raw payloads on the per-record marshal/unmarshal hot path — the hottest path in the whole system (every record crosses it, twice: source→engine and engine→destination).

Numbers (Apple M3 Max, single-record batches, -benchtime 2s)

Path raw structured ratio
Marshal (record → proto → bytes) ~1.3 µs, 29 allocs ~7.8 µs, 125 allocs ~5.5×
Unmarshal (bytes → proto → record) ~1.5 µs, 46 allocs ~8.5 µs, 201 allocs ~5×

Where the cost is

The overhead is isolated to the opencdc record conversion in this repo — Record.ToProto/FromProto and Change.ToProto/FromProto — specifically the map[string]anystructpb.Struct conversion for structured payloads (opencdc.StructuredData). structpb allocates a wrapper *structpb.Value per field (and recursively per nested field/element), which is why nested structured records dominate.

Direction

  • Profile ToProto/FromProto for structured payloads and reduce per-field allocations (candidates: a custom, allocation-lean map[string]any ↔ proto encoding instead of structpb; pooling; or a flatter wire representation for structured data).
  • Any change here is on a serialized/protocol-adjacent path — must preserve exact round-trip semantics (the protocol repo's round-trip tests + the new benchmarks are the guard). No wire-format break without a versioning discussion.
  • Re-run the protocol benchmarks before/after to quantify the win (the committed benchmarks make this a clean regression/benefit measurement).

Not urgent, but it's the single highest-leverage micro-optimization in the data path — it benefits every structured-record pipeline.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions