Adapt the socket receive buffer size to traffic

[lpgauth]

Calling recv with Length 0 reads into a fixed 8 KB buffer unless
{otp, rcvbuf} is set, so bulk transfers drain in 8 KB sips. Now a read
that fills the buffer doubles it (capped at 256 KB) and it halves back
towards the default once an EWMA of the read sizes drops below a quarter
of the buffer. Only stream sockets adapt; dgram sockets keep reading into
the configured size so truncation stays predictable, and
recvfrom/recvmsg/recvmmsg are unaffected.

Setting rcvbuf explicitly pins the size like before, since the configured
value bounds what a Length 0 recv may return, and getopt still reports
the configured value rather than the adapted one.

Draining a local bulk stream went from 4.2 to ~7.5 GB/s (M2 Pro). Caps
past 256 KB measured slower (~6.0 GB/s at 1 MB), matching the review
comments about diminishing returns.

[github-actions]

CT Test Results

    4 files    199 suites   1h 55m 10s ⏱️
3 388 tests 2 912 ✅ 474 💤 2 ❌
4 289 runs  3 736 ✅ 551 💤 2 ❌

For more details on these failures, see this check.

Results for commit a582f34.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[lpgauth]

This patch has been running in production for several days with no issues. I can share more benchmark numbers if that's useful.

[NelsonVides]

A bunch of comments for improvements but generally, it makes me think, shouldn't the caller know what to request the len to be to, if it knows it's going to fetch something bigger? The caller can be responsible for adapting the request sizes dynamically, automatic adaptation introduces a ton of complications... 😕

[NelsonVides]

nitpicking, but, I believe C refuses to reorder struct fields, so this would waste memory to padding because of their sizes: boolean is a macro to an int, and both int and unsigned int usually have 32bit sizes, and size_t usually has 64bits. Likely that

    size_t             rBufSzCfg;
    BOOLEAN_T          rBufAdapt;
    unsigned int       rBufShrinkCnt;

will be more optimal and save 4 bytes of padding.

[lpgauth]

Fixed by the rework: rBufSzCfg is gone and the remaining fields moved into the existing non-Windows block, size_t fields first.

[NelsonVides]

There's a potential issue here. I think you've checked this against TCP sockets, which behave differently than UDP ones:

• TCP (SOCK_STREAM): a short read leaves the remainder in the kernel's receive queue; the next recv picks it up. Nothing is lost, byte order is preserved, and the only thing rBufSz controls is how many syscalls it takes to drain a given number of bytes.
• UDP (SOCK_DGRAM): the datagram is consumed from the queue either, recv() returns at most one datagram; if your buffer is smaller, you get the first rBufSz bytes and the tail is discarded. There is no next read that recovers it.

So there's an issue with the adapting buffer sizes: if we receive packets becoming larger and larger until this is 1MB, but then a bunch of smaller packets arrive until this shrinks and then a larger packet arrives just after shrinking, then the larger packet is truncated, and we never knew about it.

It's an edge case, in reality UDP packets are rarely ever larger than the typical MTU of 1500 bytes. But still for correctness case, in case this is deployed in a network that has jumbo frames or whatnot, I wouldn't want the code to do surprising behaviour, previously it was predictable truncation, now is a lot more random.

Suggested change

	if ((len == 0) && descP->rBufAdapt) {
	if ((len == 0) && descP->rBufAdapt && (descP->type == SOCK_STREAM)) {

Note that only recvmsg/recvmmsg report whether a message was truncated if we give it such flag, recv/recvfrom will not report truncation. But those that can report it will potentially use buffers written here.

[lpgauth]

Good catch. Adaptation is now gated on SOCK_STREAM, both when picking the buffer size and when updating it, so dgram sockets read into the configured size like before.

[NelsonVides]

A dgram socket bulk-reads via socket:recv(S, 0) until rBufSz hits the 1 MB cap, then calls socket:recvmmsg(S, 1024, 0, 0, [], infinity)... and bufdata_sz = 1024 × 1 MB = 1 GiB. ESOCK_ASSERT is unconditional, so failing to allocate 1GB aborts the emulator rather than returning an error.

The problem is that before the PR, rBufSz had exactly one meaning: the configured default receive buffer size. Seven call sites read it that way, getopt returned it, setopt set it, accept inherited it, recv/recvfrom/recvmsg/recvmmsg used it. This PR kept the name and added a second meaning: the current adaptive working size: mutated by one site only. rBufSzCfg is introduced to carry meaning A, but only partially.

My proposal, swap them back:

• rBufSz keeps its original meaning: the configured default. What getopt returns, what setopt sets, what every receive path uses for Length == 0. Nothing else writes it.
• rBufSzCfg is deleted, replaced by rBufSzAdapt: the adaptive working size. Written in one place, read in one place, both inside essio_recv.

That way this adapting value does not interfere with the other three receiving calls, and no changes are needed to the getopt/setopt functions, making this PR also smaller.

[lpgauth]

Went with your proposal. rBufSz is back to meaning only the configured size, and the working size lives in rBufSzAdapt, which is only touched by essio_recv. The getopt/setopt changes are reverted and recvmmsg always sizes from the configured value.

[NelsonVides]

Another comment about this size is whether 1MB is the right cap. I'd assume the following formula:

time/byte      =      memcpy + syscall_cost/bufsz

So incrementing the bufsz amortises the cost of the syscall but increases the cost of the memcpy, and there's probably a threshold where the syscall's proportion of the total time is so small that trying to reduce it further barely has an impact, while the memcpy is getting more expensive. So the improvement graph will have a diminishing-returns effect, where at some point increasing the buffer more and more will give you less and less improvement. And I'm guessing 1MB is past the knee of that graph, I'd shoot at 64KB or 128KB being the sweet-spot so maybe 256KB is a safer maximum.

[lpgauth]

Benchmarked draining a local bulk stream (4 GiB over loopback, M2 Pro): 128 KB and 256 KB caps both land around 7.4-7.6 GB/s while 1 MB does ~6.0, so you were right about the knee. Capped at 256 KB.

[NelsonVides]

This makes me think that this buffer will essentially be really hard to ever shrink, it takes 32 consecutive reads under a quarter of the buffer size to ever shrink, so on a mixed-sized packets connection it might virtually never trigger. It also means an idle socket would not shrink either, and once you have 10k sockets not shrinking you have 10GB of RAM wasted.

[lpgauth]

Replaced the counter with an EWMA of the read sizes, see the other thread. On the idle point: the descriptor buffer is only kept when the last read used less than 75% of it, otherwise recv_create_bin hands it off to the returned binary, so an idle socket pins at most one buffer, now at most 256 KB. Freeing it on shrink could be a follow-up.

[essen]

A bunch of comments for improvements but generally, it makes me think, shouldn't the caller know what to request the len to be to, if it knows it's going to fetch something bigger? The caller can be responsible for adapting the request sizes dynamically, automatic adaptation introduces a ton of complications... 😕

Cowboy and parts of RabbitMQ have been using a similar strategy to this for a while with similar observations to performance improvements. My key takes from experimenting with dynamic buffer sizes are:

• You sometimes think you know how much data is coming in, but mostly you don't. Framed protocols do many things concurrently so even if you expect a large amount of data to come in, it may not come in immediately. Similarly, if you think there's no going to be much data coming in because the current message is small, you may not anticipate a flood of messages or other.

• You never know when data will stop coming in or at least slow down. Protocols don't inform you of that. Maybe you can infer it when you're the client of a request/response protocol if you have no pending requests, but not much else. So most of the time you have no trigger to reduce the buffer size and save memory.

The only good metric to decide increasing/decreasing buffer sizes is the amount of incoming data received. It can be done at the application level (like Cowboy) but if it can be done directly in the socket code it's a win for everyone (applications, distribution, etc.). And in my experience there's no real drawback to doing this. In Cowboy/RabbitMQ's case it helps both save memory (because the base buffer size is smaller) and increase performance, win win.

[essen]

I would recommend allowing configuration of low/high bounds for the dynamic buffer size. Cowboy uses 1K/128K and it works well, saving memory when idle, not just improving performance otherwise.

[NelsonVides]

Putting a condition on len == 0 also means that if this read could have contributed to a shrink because it got much less, it won't contribute. Perhaps that's a good thing, dunno, I'm just concerned about this logic eventually never shrinking.

[lpgauth]

With the EWMA this matters less, but I kept explicit-length reads out of the accounting on purpose: their size is bounded by the caller's length, so they say nothing about how much a length 0 read would have gotten.

[NelsonVides]

One option for the shrinking never happening is for this condition not to reset the counter to zero. Say we reached the 1MB cap, and then forever after reads will have 255KB and one every 32 reads will have 257KB, with the exisiting logic we'll never shrink. I'd make this

Suggested change

	descP->rBufShrinkCnt = 0;
	descP->rBufShrinkCnt--;

Another option, instead of using consecutive counters as currently, or a leaky counter as just suggested above, would be to use an EWMA, that usually has super simple code and very good converging around stable rates and resilience against outliers.

[lpgauth]

Went with the EWMA (alpha 1/8, shift arithmetic). Growth is still doubling on a filled read so the ramp-up is unchanged, and shrink halves the buffer whenever the average falls below a quarter of it. A one-off big read among small ones now shrinks back within a few reads instead of ratcheting.

[NelsonVides]

Hmm... I see. Thing is this is sufficiently complicated to prefer application code just passing its own dynamically calculated len values, but being able to benefit everyone for free is also a win. But that means the shrinking logic needs more power, the current logic is too simple, might virtually never free, the upper bound is too high...

So three important points:

• Set a right cap at just 2x the knee in the diminishing returns function (when the weight of memcpy starts to balance out against the less frequent syscalls, the point where until then every buffer growth gave big improvements and thereafter duplicating the buffer barely improves a percentage point). I'd guess that knee is around 64KB or 128KB so a good max is 256KB, not 1MB.
• Ensure the shrinking logic is statistically reachable: currently once the buffer reached 1MB, if all future inputs are uniformly distributed between 100KB and 200KB, it will take around 90000 recv calls to reach a 256KB buffer. Use a leaky counter or a EWMA.
• Ensure the logic is compatible with all four receive calls: recv/recvfrom/recvmsg/recvmmsg. I know, I was the one that added recvmmsg and it took me forever to not get confused between all of them 😄

[lpgauth]

Pushed a rework as separate commits for easier re-review, will squash once the review settles.

For context, the inet driver already adapts its read buffer to the traffic, so gen_tcp on the default backend gets this for free. The goal here is to close that gap and make the socket backend competitive with inet for bulk transfers.

rBufSz is back to only meaning the configured size and the adaptive size is a separate field that only essio_recv touches, so recvfrom/recvmsg/recvmmsg and getopt/setopt behave exactly as before. Adaptation only applies to stream sockets now. The shrink counter is replaced with an EWMA of the read sizes, and the cap is down to 256 KB after benchmarking (128 KB and 256 KB drain at ~7.5 GB/s on my M2 Pro vs ~6.0 for 1 MB). Also added a test to socket_api_SUITE covering growth, pinning via {otp, rcvbuf} and dgram non-adaptation.

@essen configurable bounds make sense, but they need a decision on the option shape ({otp, rcvbuf} already uses the 2-tuple for {N, BufSz} and gen_tcp_socket matches on the existing shapes), so I'd rather do that as a follow-up once this lands.