Uncompressed binary inserts emit one HTTP chunk per field write, inflating the request body 25-70%

[alex-clickhouse]

Summary

When a binary insert is sent uncompressed (InsertOptions.Compressor = null), the driver emits
one HTTP chunk per Write call — roughly one per field, per row. On a 1-column Int64 table that
is 1001 chunks of 8 bytes each for 1000 rows, and the request body ends up 70% larger than the data
it carries. On a realistic 15-column schema the inflation is ~25%.

Compressed inserts are unaffected, which is why this has stayed invisible: every compressor wraps
itself in a 256 KiB PooledWriteBufferStream, so its writes are coalesced. The uncompressed path is
the only one with no buffer between ExtendedBinaryWriter and the HTTP request stream.

Measured

Captured by pointing InsertBinaryAsync at a TCP sink that records the raw request bytes, so these
are wire bytes including chunk framing, not an estimate. One Int64 column:

rows	Compressor	payload	wire bytes	chunks	median chunk
100	null	800 B	1,869 B	101	8 B
100	lz4	800 B	1,028 B	4	—
1000	null	8,000 B	13,569 B	1001	8 B
1000	lz4	8,000 B	4,626 B	4	—

The per-row cost is the chunk framing itself: "8\r\n" + 8 payload bytes + "\r\n" = 13 wire bytes
to carry 8 bytes of data, so ~5 B/row plus one chunk for the query line.

On a 15-column hits subset (WatchID Int64, JavaEnable Int16, Title String, EventTime DateTime, CounterID Int32, ClientIP Int32, RegionID Int32, UserID Int64, URL String, Referer String, IsRefresh Int16, ResolutionWidth Int16, SearchPhrase String, RefererHash Int64, URLHash Int64), 100k rows:

• SELECT … FORMAT RowBinary of the same rows — the true payload size — is 36,812,926 B
• the driver's uncompressed insert reports ProfileEvents['NetworkReceiveBytes'] = ~46.0 MB, i.e.
  +25%, about +92 B/row

That +92 B/row matches the mechanism exactly: this schema issues 19 Write calls per row (15 fields,
of which 4 are strings that write the length prefix and the bytes separately), and 19 × ~5 B ≈ 92 B.

NetworkReceiveBytes was checked against a request of known size first — a raw 36,812,926-byte body
is reported as exactly 36812926 — so the metric is not itself adding the overhead.

Cause

ClickHouse.Driver/Copy/Serializer/BatchSerializer.cs:32-37:

var compressing = compressor != null;
var target = compressing ? compressor.Compress(stream, leaveOpen: true) : stream;

PooledStreamWriter.WriteLine(target, batch.Query);

using var writer = new ExtendedBinaryWriter(target, leaveOpen: !compressing);

When compressing, compressor.Compress(...) returns a PooledWriteBufferStream wrapping the codec
(GZipCompressor.cs:37, Lz4Compressor.cs:60, BrotliCompressor.cs:40), so writes accumulate in a
256 KiB buffer and reach the transport in large blocks. When not compressing, target is the
request stream, and the body is sent as StreamContent (ClickHouseClient.cs:875) — an unknown-length
stream, so Transfer-Encoding: chunked. Every small Write therefore becomes its own chunk.

Suggested fix

Give the uncompressed path the same buffering the compressed path already gets — conceptually:

var target = compressing
    ? compressor.Compress(stream, leaveOpen: true)
    : new PooledWriteBufferStream(stream, bufferSize, leaveOpen: true);

Two things to watch:

1. Flush ordering. The current leaveOpen: !compressing exists so the caller can seek/read the
   stream afterwards (per the comment at BatchSerializer.cs:29-31). A buffer has to be flushed
   before that read, or the tail of the batch is lost.
2. Visibility. PooledWriteBufferStream is internal to ClickHouse.Driver.Common, and that
   assembly's only InternalsVisibleTo is ClickHouse.Driver.Tests — not ClickHouse.Driver. So
   this needs either an added InternalsVisibleTo, or a plain BufferedStream, which would do the
   job here.

Impact and scope

• InsertOptions.Compressor defaults to GZipCompressor.Default, so the default insert path is not
  affected. This only bites callers who explicitly set Compressor = null.
• That is still a case worth fixing: opting out of compression is the reasonable choice on a fast or
  loopback network, or when the payload is already incompressible — and it is precisely there that a
  25–70% wire increase, plus a chunk (and a write syscall) per field, is most surprising.
• Pre-existing on main; not introduced by any in-flight compression work. Found while benchmarking
  codecs for feat(compression): vendor ZstdSharp and add a built-in ZstdCompressor (no new dependency) #523, whose changes do not touch BatchSerializer.

Environment: driver main-based branch, net10.0, ClickHouse 26.6.1.1193 over HTTP.