High CPU usage when sending data to broken socket connection on Linux

[mumez]

Description

The VM shows extremely high CPU usage when data is sent to a socket and the client connection is destroyed. This issue is specific to Linux VM and does not occur on Windows.

Key characteristics:

• High CPU usage starts immediately when there is data sent but not received AND the client socket is closed
• The order of close/send operations doesn't matter
• Issue is Linux-specific (not reproducible on Windows VM)
• This is a regression - the issue did not occur with VM version 10.0.5
• Confirmed to occur with VM versions 10.3.6 and later

Code to Reproduce

Minimal reproduction case:

server := Socket newTCP listenOn: 12345 backlogSize: 5.
[client := Socket newTCP connectTo: NetNameResolver localHostAddress port: 12345] fork.
server waitForConnectionFor: 30.
accepted := server accept.
accepted inspect. "OK"

"The following order is irrelevant if these are both done"
accepted sendData: (String new: 1 withAll: $a). 
client destroy. 

Target VM

• Platform: Linux
• Affected versions: 10.3.6 and later
• Working version: 10.0.5
• Not affected: Windows VM

Expected Behavior

The VM should handle broken socket connections gracefully without consuming excessive CPU resources, similar to the behavior in VM version 10.0.5 and current Windows VM behavior.

Actual Behavior

Extremely high CPU usage when attempting to send data to a broken socket connection

Additional Context

This issue simulates real-world scenarios where socket connections are unexpectedly broken (e.g., web browser closing unexpectedly). The regression between VM versions 10.0.5 and 10.3.6 suggests this was introduced by changes in socket handling.

Split from the issue:
#939

Screenshot

[daniels220]

False alarm—at least as far as this issue is concerned—on the crash-on-GC, it looks like that actually happens on my Linux machine even without any sockets stuff, so I have to dig into that more. (I assume, @mumez, that you removed that from the title after being unable to reproduce for yourself.) But I do have some updates that are relevant here.

I tried a couple other things to rule out weird edge-cases—no smoking guns unfortunately, but then, there really shouldn't be:

• Sending data using the client socket and closing the accepted socket behaves the same—makes sense, once connected the pair should be totally symmetric, but I thought I'd check.

  "Or this will also work, again in either order"
  client sendData: (String new: 1 withAll: $a). 
  accepted destroy.

• Having one end of the connection be an actual separate program on an actual separate machine behaves the same. This is true regardless of which is the "client" and which is the "server".
• In the above case of separate programs/images, only the image that is trying to send data to a closed socket has the problem, not the one that forcibly closed the socket. So we can infer (as more-or-less expected) that the problem is with having a socket that has sent-but-not-ACK'ed data and the other end is closed.

I also had the thought of, what happens if I close the other end of the connection (accepted destroy), so the whole thing should be fully closed? And what I found was honestly the one thing I did not expect—CPU usage drops, but remains high, about 15-20% on my machine where it had been <5% at idle before. I tried destroying the original listening socket (server destroy) as well, and that made no further difference.

I tried a crude test to make sure the problem is not occurring in the image somehow:

[samples := IdentityDictionary new.
1000 timesRepeat: 
		[| proc |
		Processor sleep: 1.
		proc := Processor preemptedProcess.
		samples at: proc put: (samples at: proc ifAbsent: [0]) + 1].
samples associations inspect]
		forkAt: Processor lowIOPriority.

and I got >500 samples hitting the idle process—so pretty clearly, no, the image thinks it's mostly idle, but the VM is freaking out.

In case it hasn't been stated yet, note that trying to send more data from the side of the connection that is not destroyed raises an error, as expected (connection closed while sending data), but this makes no difference to the high CPU load.

[mumez]

I created a repository to easily test the issue.
https://github.com/mumez/pharo-linux-vm-docker-load-test/tree/main

I will test to see if the issue is still reproduced with the latest 10.3.9 vm.

[mumez]

Unfortunately, the issue is still present in the latest version, 10.3.9. I put a test report here.
https://github.com/mumez/pharo-linux-vm-docker-load-test/blob/main/doc/TestResults.md

@tesonep I think you can easily reproduce the issue by simply cloning the repo and running the test scripts.

GitHub

pharo-linux-vm-docker-load-test/doc/TestResults.md at main · mumez/pharo-linux-vm-docker-load-test

A Docker-based testing environment for evaluating CPU load of specific Pharo VM versions on Linux - mumez/pharo-linux-vm-docker-load-test

[daniels220]

I can confirm, the issue is still present in the 10.3.9 Linux VM. It does seem like it must be caused by something in the UNIX-specific parts of either the socket plugin or the AIO implementation.