one zmq.PUSH socket permanently blocks on send_pyobj() while other PUSH sockets to the same PULL continue working

[937869301]

We encountered this issue while running SGLang. Multiple scheduler processes send messages to a single tokenizer process using ZeroMQ.

Environment:

libzmq 4.3.5
pyzmq 27.1.0
TCP transport (tcp://127.0.0.1:)
8 independent PUSH sockets (one per process/rank) connected to a single PULL socket.

The sender is created as:

socket = context.socket(zmq.PUSH)
socket.setsockopt(zmq.SNDHWM, 0)
socket.setsockopt(zmq.SNDBUF, buf_size)
socket.connect(endpoint)

The receiver is:

socket = context.socket(zmq.PULL)
socket.setsockopt(zmq.RCVHWM, 0)
socket.setsockopt(zmq.RCVBUF, buf_size)
socket.bind(endpoint)

The send path is simply:

events_before = self.socket.getsockopt(zmq.EVENTS)
logger.info("before send events=%x", events_before)

self.socket.send_pyobj(output)

events_after = self.socket.getsockopt(zmq.EVENTS)
logger.info("after send events=%x", events_after)

Very rarely, all 8 processes enter send_pyobj(), but the PULL socket only receives 7 messages. One PUSH socket blocks until SNDTIMEO expires and raises zmq.Again.

After the first failure, every subsequent send_pyobj() on that same PUSH socket also times out, while the other 7 PUSH sockets continue sending to the same PULL socket normally.

Before the failure, getsockopt(zmq.EVENTS) returns 2 (POLLOUT). After the failure, it consistently returns 0, and the affected sender never recovers unless the socket/process is recreated.

[minrk]

@bluca this question about libzmq socket behavior was erroneously transferred to pyzmq, so I'm migrating it back to libzmq. pyzmq is not the place for libzmq socket behavior questions, even when they happen to be accessed via python bindings. A libzmq PUSH socket is blocking, and if the description is accurate that getsockopt(ZMQ_EVENTS) is returning POLLOUT, but send on that socket is still blocking, I think that would indeed be a libzmq bug.