Catheter action for RL

[MarkWrobelUCL]

Hi, I am trying to use sofa-gym in order to train an agent to move a catheter from one area of a model heart to another. Previously I have been using imitation learning techniques which have been relatively successful. My scene is based on this scene in the beam adapter example. This uses 'InterventionalRadiologyController' to move the catheter. Now I am following the CartPole example and am unsure how to implement the class 'ApplyAction' for my purposes, in order to move the catheter. Any help would be appreciated. Thanks, Mark.

[samuelmyoussef]

Hello,

Your scene is probably similar to the CTR environment in SofaGym. You can look at the implementation here to help you, specifically the startCmd, start_cmd, and action_to_command functions. You don't need to explicitly implement the ApplyAction class but it helps with making your code more organized. The main difference between the CTR env and the scene you mentioned is that the CTR controls 3 instruments instead of one in the case of the single beam example.

We will also add a new example of a catheter soon, based on the 3instrument scene from beamadapter, which is similar to your scene but with 3 concentric tubes instead of a single one, so keep an eye on this.

[MarkWrobelUCL]

Thank you so much for the help @samuelmyoussef . I now have something working but I still have a few questions. In the CTREnv.py in line 61 there is the variable dim_state and I'm not quite sure what it means. I understand why there are 12 actions in this example, but not what dim_state represents. Also it appears the goal is randomized, does this only happen when you start to train i.e. it is fixed for the entirety of each training attempt?

Also do you happen to know what the variable xtip in the InterventionalRadiologyController represents?

Again many thanks for the help.

[samuelmyoussef]

dim_state represents the dimension of the observation or the state of the environment that is returned after each step. In this case it's 12 because the observation includes the xtips and rotations of the 3 instruments, the xyz position of the beam's tip, and the xyz position of the goal. You can see it in the code here returned by the getState method.

https://github.com/SofaDefrost/SofaGym/blob/main/sofagym/envs/CTR/CTRToolbox.py#L216

The goal is randomized for each episode not for the whole training attempt, so when an episode ends and the environment is reset, the goal will be different for the new episode.

For xtip, as given by the description of the variable it's the curvilinear abscissa of the tip of each interventional radiology instrument, so basically a measure of the curvature of each of the instruments of the catheter.

[MarkWrobelUCL]

Thanks for the explanations @samuelmyoussef . I now have something that trains well (it is quite a simple task). The catheter consistently gets to the goal position within the heart during training and is also using very few actions to get there. This sounds great however when I try to evaluate the trained agent, it just rotates at the start point which is not correct. Do you have any idea why the agent is very good at consistently hitting the maximum reward during training but then not when testing the trained agent? It might be useful to also mention that the simulation of the agent between epochs never hits the goal position either even though the agent is consistently hitting that same goal during the epoch. The goal i have set is the same across all epochs of training.

If you have any ideas about this issue I would be grateful. Thanks once more!

[samuelmyoussef]

We are aware of this issue and it's been fixed in this PR #45 but it hasn't been merged to the main branch yet. Until then, you can try to use my branch from the PR or apply the modifications manually to see if it works and give us your feedback.

[MarkWrobelUCL]

Thanks for letting me know. When I try to run any RL training (even the examples) with this branch I get a bunch of errors. Now I have tried with this version from Aug 28, the RL runs however, it gets stuck at 99/1000 during epoch 1 even for the example environments.

Please advise if I am doing something wrong. Thanks

[samuelmyoussef]

Can you post the errors log, please?

[MarkWrobelUCL]

Here is the error log for the up to date branch:

`[ERROR] [SofaRuntime] AssertionError: The render_mode must be 'rgb_array', not None
Traceback (most recent call last):
File "/home/mark/Documents/SofaGym-fix_model_load/rl.py", line 113, in
agent.fit(total_timesteps)
File "/home/mark/Documents/SofaGym-fix_model_load/agents/SB3Agent.py", line 309, in fit
self.model.learn(total_timesteps=total_timesteps, reset_num_timesteps=False,
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/ppo/ppo.py", line 308, in learn
return super().learn(
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/on_policy_algorithm.py", line 246, in learn
total_timesteps, callback = self._setup_learn(
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 436, in _setup_learn
callback = self._init_callback(callback, progress_bar)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 380, in _init_callback
callback.init_callback(self)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback
self._init_callback()
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 194, in _init_callback
callback.init_callback(self.model)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback
self._init_callback()
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 194, in _init_callback
callback.init_callback(self.model)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 155, in init_callback
self.callback.init_callback(self.model)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 63, in init_callback
self._init_callback()
File "/home/mark/Documents/SofaGym-fix_model_load/agents/callbacks.py", line 153, in _init_callback
self.eval_env = VecVideoRecorder(self.eval_env, self.save_path,
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 50, in init
assert self.env.render_mode == "rgb_array", f"The render_mode must be 'rgb_array', not {self.env.render_mode}"

########## SIG 15 - SIGTERM: a termination request was sent to the program ##########

########## SIG 15 - SIGTERM: a termination request was sent to the program ##########
sofa::helper::BackTrace::sig(int)
read
_Py_read
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyObject_Call
_PyEval_EvalFrameDefault
PyEval_EvalCode
PyRun_StringFlags
PyRun_SimpleStringFlags
Py_RunMain
Py_BytesMain
__libc_start_main
sofa::helper::BackTrace::sig(int)
read
_Py_read
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyObject_Call
_PyEval_EvalFrameDefault
PyEval_EvalCode
PyRun_StringFlags
PyRun_SimpleStringFlags
Py_RunMain
Py_BytesMain
__libc_start_main
[ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in
task = s.getNextTask(stateId)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call
return self.__send(self.__name, args)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in __request
response = self.__transport.request(
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request
self.send_content(connection, request_body)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content
connection.endheaders(request_body)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect
self.sock = self._create_connection(
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection
raise err
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection
sock.connect(sa)

[ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in
task = s.getNextTask(stateId)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call
return self.__send(self.__name, args)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in __request
response = self.__transport.request(
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request
self.send_content(connection, request_body)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content
connection.endheaders(request_body)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect
self.sock = self._create_connection(
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection
raise err
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection
sock.connect(sa)

Exception ignored in: <function VecVideoRecorder.del at 0x7f9a5ff6ce50>
Traceback (most recent call last):
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in del
self.close_video_recorder()
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 103, in close_video_recorder
if self.recording:
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 392, in getattr
return self.getattr_recursive(name)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive
attr = self.venv.getattr_recursive(name)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive
attr = self.venv.getattr_recursive(name)
File "/home/mark/.local/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 417, in getattr_recursive
attr = getattr(self.venv, name)
AttributeError: 'SubprocVecEnv' object has no attribute 'recording'`

[samuelmyoussef]

Seems you are using a newer version of stable baseline 3, which doesn't work with SofaGym for now. Can you try with version 1.7.0

[MarkWrobelUCL]

Thank you, this has fixed the errors. However, the training still stops at 99/1000 during epoch 1. There are no error messages. The environment simply resets every so often but remains on 99/1000.

[samuelmyoussef]

If you left the hyperparameters in the yaml file as they are, then it's running 5 evaluations every 100 steps, so it's probably just running this evaluation callback not stuck so it should continue after some time if you leave it running. You should change the eval_freq for whichever algorithm you're using if you want the evaluation to happen less frequently.

[MarkWrobelUCL]

I have left it running for longer now and I indeed do get an error after about 3 minutes.

`[ERROR] [SofaRuntime] EOFError
Traceback (most recent call last):
File "/home/mark/Documents/SofaGym-fix_model_load/rl.py", line 113, in
agent.fit(total_timesteps)
File "/home/mark/Documents/SofaGym-fix_model_load/agents/SB3Agent.py", line 309, in fit
self.model.learn(total_timesteps=total_timesteps, reset_num_timesteps=False,
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/dqn/dqn.py", line 265, in learn
return super().learn(
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 334, in learn
rollout = self.collect_rollouts(
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/off_policy_algorithm.py", line 575, in collect_rollouts
if callback.on_step() is False:
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step
return self._on_step()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step
continue_training = callback.on_step() and continue_training
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step
return self._on_step()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 204, in _on_step
continue_training = callback.on_step() and continue_training
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 100, in on_step
return self._on_step()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/callbacks.py", line 447, in _on_step
episode_rewards, episode_lengths = evaluate_policy(
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/evaluation.py", line 89, in evaluate_policy
observations, rewards, dones, infos = env.step(actions)
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 163, in step
return self.step_wait()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 76, in step_wait
obs, rewards, dones, infos = self.venv.step_wait()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_normalize.py", line 175, in step_wait
obs, rewards, dones, infos = self.venv.step_wait()
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 121, in step_wait
results = [remote.recv() for remote in self.remotes]
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 121, in
results = [remote.recv() for remote in self.remotes]
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError

########## SIG 15 - SIGTERM: a termination request was sent to the program ##########
sofa::helper::BackTrace::sig(int)
read
_Py_read
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyObject_Call
_PyEval_EvalFrameDefault
PyEval_EvalCode
PyRun_StringFlags
PyRun_SimpleStringFlags
Py_RunMain
Py_BytesMain
__libc_start_main
[ERROR] [SofaRuntime] ConnectionRefusedError: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/mark/Documents/SofaGym-fix_model_load/sofagym/rpc_client.py", line 181, in
task = s.getNextTask(stateId)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1122, in call
return self.__send(self.__name, args)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1464, in __request
response = self.__transport.request(
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1178, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1291, in send_request
self.send_content(connection, request_body)
File "/home/mark/miniconda3/lib/python3.10/xmlrpc/client.py", line 1321, in send_content
connection.endheaders(request_body)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 1037, in _send_output
self.send(msg)
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 975, in send
self.connect()
File "/home/mark/miniconda3/lib/python3.10/http/client.py", line 941, in connect
self.sock = self._create_connection(
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 845, in create_connection
raise err
File "/home/mark/miniconda3/lib/python3.10/socket.py", line 833, in create_connection
sock.connect(sa)

Exception ignored in: <function VecVideoRecorder.del at 0x7f34331bbb50>
Traceback (most recent call last):
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in del
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 109, in close
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 279, in close
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/vec_monitor.py", line 100, in close
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 279, in close
File "/home/mark/miniconda3/lib/python3.10/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 144, in close
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 250, in recv
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
File "/home/mark/miniconda3/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
EOFError:
Exception ignored in: <function tqdm.del at 0x7f348b5f67a0>
Traceback (most recent call last):
File "/home/mark/.local/lib/python3.10/site-packages/tqdm/std.py", line 1145, in del
File "/home/mark/.local/lib/python3.10/site-packages/tqdm/rich.py", line 120, in close
File "/home/mark/.local/lib/python3.10/site-packages/rich/progress.py", line 1177, in exit
File "/home/mark/.local/lib/python3.10/site-packages/rich/progress.py", line 1163, in stop
File "/home/mark/.local/lib/python3.10/site-packages/rich/live.py", line 155, in stop
File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1136, in line
File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1673, in print
File "/home/mark/.local/lib/python3.10/site-packages/rich/console.py", line 1534, in _collect_renderables
File "/home/mark/.local/lib/python3.10/site-packages/rich/protocol.py", line 28, in rich_cast
ImportError: sys.meta_path is None, Python is likely shutting down`