Help understanding server issue

[ebaldelomar]

Hello,

I've been trying to set up MONAILabel and 3D slicer on one of our institution servers to take advantage of its high GPU instead of using a local machine.

With the help of our IT, we got MONAILabel installed starting from the prerequisites following instructions (https://docs.monai.io/projects/label/en/latest/installation.html). Now I'm trying to follow the case 2 example in the Quickstart documentation (https://docs.monai.io/projects/label/en/latest/quickstart.html#use-case-2-bundle-with-customized-scripts-for-renal-substructure-segmentation).

This ran overnight and I think is stuck. Below are the log outputs.

[nmr141@bmrs116 ~]$ monailabel start_server --app monaibundle --studies Task09_Spleen/imagesTr --conf models renalStructures_UNEST_segmentation
Using PYTHONPATH=/bmr207/nmrgrp/nmr141:

/bmr207/nmrgrp/nmr141/.local/lib/python3.8/site-packages/ignite/handlers/checkpoint.py:17: DeprecationWarning: TorchScript support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the torch.compile optimizer instead.
from torch.distributed.optim import ZeroRedundancyOptimizer
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: version = False
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: app = /bmrc-homes/nmrgrp/nmr141/monaibundle
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: studies = /bmrc-homes/nmrgrp/nmr141/Task09_Spleen/imagesTr
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: verbose = INFO
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'renalStructures_UNEST_segmentation']]
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: port = 8000
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None
[2025-04-07 16:07:59,499] [3155126] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: workers = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: access_log = False
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: root_path = /
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: log_level = info
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: log_config = None
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: dryrun = False
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:285) - USING:: action = start_server
[2025-04-07 16:07:59,500] [3155126] [MainThread] [INFO] (main:296) -
Allow Origins: ['*']
[2025-04-07 16:08:03,436] [3155126] [MainThread] [INFO] (uvicorn.error:82) - Started server process [3155126]
[2025-04-07 16:08:03,436] [3155126] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup.
[2025-04-07 16:08:03,436] [3155126] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: /bmrc-homes/nmrgrp/nmr141/monaibundle; studies: /bmrc-homes/nmrgrp/nmr141/Task09_Spleen/imagesTr; conf: {'models': 'renalStructures_UNEST_segmentation'}
[2025-04-07 16:08:03,920] [3155126] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2025-04-07 16:09:02,916] [3155126] [MainThread] [INFO] (monailabel.utils.others.generic:315) - +++ Adding Bundle from Zoo: renalStructures_UNEST_segmentation => /bmrc-homes/nmrgrp/nmr141/monaibundle/model/renalStructures_UNEST_segmentation
2025-04-07 16:09:02,917 - INFO - --- input summary of monai.bundle.scripts.download ---
2025-04-07 16:09:02,917 - INFO - > name: 'renalStructures_UNEST_segmentation'
2025-04-07 16:09:02,917 - INFO - > bundle_dir: '/bmrc-homes/nmrgrp/nmr141/monaibundle/model'
2025-04-07 16:09:02,917 - INFO - > source: 'github'
2025-04-07 16:09:02,917 - INFO - > repo: 'Project-MONAI/model-zoo/hosting_storage_v1'
2025-04-07 16:09:02,917 - INFO - > remove_prefix: 'monai_'
2025-04-07 16:09:02,917 - INFO - > progress: True
2025-04-07 16:09:02,917 - INFO - ---

renalStructures_UNEST_segmentation_v0.2.2.zip: 309MB [00:15, 20.3MB/s]
2025-04-07 16:09:28,488 - INFO - Downloaded: /bmrc-homes/nmrgrp/nmr141/monaibundle/model/renalStructures_UNEST_segmentation_v0.2.2.zip
2025-04-07 16:09:28,488 - INFO - Expected md5 is None, skip md5 check for file /bmrc-homes/nmrgrp/nmr141/monaibundle/model/renalStructures_UNEST_segmentation_v0.2.2.zip.
2025-04-07 16:09:28,488 - INFO - Writing into directory: /bmrc-homes/nmrgrp/nmr141/monaibundle/model.
[2025-04-07 16:09:33,363] [3155126] [MainThread] [INFO] (monailabel.utils.others.generic:375) - +++ Using Bundle Models: ['renalStructures_UNEST_segmentation']
[2025-04-07 16:09:33,364] [3155126] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: /bmrc-homes/nmrgrp/nmr141/Task09_Spleen/imagesTr
[2025-04-07 16:09:33,365] [3155126] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml']
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_56 => spleen_56.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_6 => spleen_6.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_62 => spleen_62.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_2 => spleen_2.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_3 => spleen_3.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_31 => spleen_31.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_24 => spleen_24.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_13 => spleen_13.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_14 => spleen_14.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_44 => spleen_44.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_27 => spleen_27.nii.gz
[2025-04-07 16:09:33,375] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_52 => spleen_52.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_17 => spleen_17.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_46 => spleen_46.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_26 => spleen_26.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_9 => spleen_9.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_16 => spleen_16.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_21 => spleen_21.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_49 => spleen_49.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_22 => spleen_22.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_41 => spleen_41.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_40 => spleen_40.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_29 => spleen_29.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_38 => spleen_38.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_32 => spleen_32.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_28 => spleen_28.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_60 => spleen_60.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_45 => spleen_45.nii.gz
[2025-04-07 16:09:33,376] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_10 => spleen_10.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_33 => spleen_33.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_19 => spleen_19.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_25 => spleen_25.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_61 => spleen_61.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_53 => spleen_53.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_20 => spleen_20.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_12 => spleen_12.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_47 => spleen_47.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_18 => spleen_18.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_8 => spleen_8.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_63 => spleen_63.nii.gz
[2025-04-07 16:09:33,377] [3155126] [MainThread] [INFO] (monailabel.datastore.local:594) - Adding New Image: spleen_59 => spleen_59.nii.gz
[2025-04-07 16:09:33,383] [3155126] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 41
[2025-04-07 16:09:33,387] [3155126] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2025-04-07 16:09:36,718] [3155126] [MainThread] [INFO] (main:63) - +++ Adding Inferer:: renalStructures_UNEST_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fa315640f10>
[2025-04-07 16:09:36,719] [3155126] [MainThread] [INFO] (main:77) - +++ Adding Trainer:: renalStructures_UNEST_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fa315b0f070>
[2025-04-07 16:09:36,719] [3155126] [MainThread] [INFO] (main:87) - Active Learning Strategies:: ['random', 'first']
[2025-04-07 16:09:36,719] [3155126] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /bmr207/nmrgrp/nmr141/.cache/monailabel/sessions
[2025-04-07 16:09:36,719] [3155126] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2025-04-07 16:09:36,719] [3155126] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed
[2025-04-07 16:09:36,720] [timeloop] [INFO] Starting Timeloop..
[2025-04-07 16:09:36,720] [3155126] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2025-04-07 16:09:36,720] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fa314f18310>
[2025-04-07 16:09:36,720] [3155126] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fa314f18310>
[2025-04-07 16:09:36,720] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2025-04-07 16:09:36,720] [3155126] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2025-04-07 16:09:36,721] [3155126] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete.
[2025-04-07 16:09:36,721] [3155126] [MainThread] [INFO] (uvicorn.error:214) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

`

In addition to this, I wanted to try using the MONAIlabel app in 3D Slicer and lick the refresh button for the server I get a 'Failed to fetch models from remote server. Make sure server address is correct and <server_uri>/info/ is accessible in browser.' error. Below is the detailed logs from that.

Traceback (most recent call last):
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/slicer.org/Extensions-33241/MONAILabel/lib/Slicer-5.8/qt-scripted-modules/MONAILabel.py", line 1100, in fetchInfo
info = self.logic.info()
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/slicer.org/Extensions-33241/MONAILabel/lib/Slicer-5.8/qt-scripted-modules/MONAILabel.py", line 2364, in info
return self._client().info()
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/slicer.org/Extensions-33241/MONAILabel/lib/Slicer-5.8/qt-scripted-modules/MONAILabel.py", line 2340, in _client
if mc.auth_enabled():
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/slicer.org/Extensions-33241/MONAILabel/lib/Slicer-5.8/qt-scripted-modules/MONAILabelLib/client.py", line 83, in auth_enabled
status, response, _, _ = MONAILabelUtils.http_method("GET", self._server_url, selector)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/slicer.org/Extensions-33241/MONAILabel/lib/Slicer-5.8/qt-scripted-modules/MONAILabelLib/client.py", line 521, in http_method
conn.request(method, selector, body=body, headers=headers)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 1285, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 1331, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 1280, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 1040, in _send_output
self.send(msg)
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 980, in send
self.connect()
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/http/client.py", line 946, in connect
self.sock = self._create_connection(
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/socket.py", line 844, in create_connection
raise err
File "/bmrc-homes/nmrgrp/nmr141/Slicer-5.8.1-linux-amd64/lib/Python/lib/python3.9/socket.py", line 832, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

One thought I had is maybe a firewall connection issue on the server at our institution? Not sure. But any guidance is very much appreciated!

Cheers,
Edwin

p.s. One thing I wasn't too keen on was the part of the instructions for a token. I did set one up through gitHub and used the following command to set it up. Hopefully this was acceptable:

set MONAI_ZOO_AUTH_TOKEN = ...(hidden for privacy)

[ebaldelomar]

This is resolved. I didn't realize one needed to run the 'monailabel' command, which would be continuously running, and then open work in 3D slicer. I thought they were subsequent steps.

Though I do have a new issue in 3D slicer that it failed to run inference (topic below). Any help is greatly appreciated.

https://discourse.slicer.org/t/failed-to-run-inference-on-inference-in-monai-label-server/42492/1