Skip to content

Fixed client hanging when using startup() after stop() a few times, and fixed is_server_connected being always True#100

Open
hello-robot-shehab wants to merge 6 commits into
mainfrom
bugfix/robot-client-zombie-zmq
Open

Fixed client hanging when using startup() after stop() a few times, and fixed is_server_connected being always True#100
hello-robot-shehab wants to merge 6 commits into
mainfrom
bugfix/robot-client-zombie-zmq

Conversation

@hello-robot-shehab

@hello-robot-shehab hello-robot-shehab commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Many hours were wasted on clients' zmq hanging for no reason. The bug is caught!

The main culprit appears to be self.context.term() unable to close out zmq sockets until Ctrl+C was pressed. We'd known this gave a zmq exception:

Traceback (most recent call last):
    File "/home/shehab/.local/lib/python3.12/site-packages/zmq/sugar/context.py", line 140, in __del__
      self.destroy()
    File "/home/shehab/.local/lib/python3.12/site-packages/zmq/sugar/context.py", line 322, in destroy
      self.term()
    File "/home/shehab/.local/lib/python3.12/site-packages/zmq/sugar/context.py", line 264, in term
      super().term()
    File "zmq/backend/cython/_zmq.py", line 596, in zmq.backend.cython._zmq.Context.term
      _check_rc(rc)
      ^^^^^^^^^^^
    File "zmq/backend/cython/_zmq.py", line 179, in zmq.backend.cython._zmq._check_rc
      PyErr_CheckSignals()
      ^^^^^^^^^^^
  KeyboardInterrupt

#94 describes a similar behavior with zmq hanging.

Steps to reproduce:

  1. The easier way it to run test/test_client_connection.py on the main branch (Note: this file is introduced in this PR, so just copy it to your main and run it!)
  2. Run it and notice that it hangs forever until you ctrl+c, which then gives the exception above.
  3. Another test that used to not work on main is the test_require_connection.py, this is fixed in this PR.

To test

  1. Run test/test_client_connection.py
  2. Run test/test_server_connection.py
  3. Run test/test_requite_connection.py

And observe that they pass.

…erver isn't running, causing hanging. Cleaned up stop() in Client classes.
@hello-robot-shehab hello-robot-shehab changed the title StretchBodyClient.startup() was not cleaning up open zmq sockets if server isn't running. StretchBodyClient.server_connected is always true, even after stop() is called. Jul 15, 2026
@hello-robot-shehab hello-robot-shehab changed the title StretchBodyClient.server_connected is always true, even after stop() is called. StretchBodyClient.server_connected was always true, even after stop() is called. Jul 15, 2026
@hello-robot-shehab hello-robot-shehab changed the title StretchBodyClient.server_connected was always true, even after stop() is called. Fixed client hanging when using startup() after stop() a few times, and fixed is_server_connected being always True Jul 15, 2026
@hello-robot-shehab
hello-robot-shehab marked this pull request as ready for review July 15, 2026 17:48
@hello-binit
hello-binit requested review from hello-binit and removed request for hello-bharadwaj July 15, 2026 19:08
@hello-robot-shehab
hello-robot-shehab marked this pull request as draft July 15, 2026 19:12
@hello-robot-shehab

hello-robot-shehab commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

I ran into the zmq bug again during REx_camera_calibate.

Okay this was happening because of gc. Added a test for garbage collection, and fixed the hanging by using linger=0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant