run_manager error

[ductrong5x5]

Hi, I working with Flare in production mode. I have this issue when submiting job. Client is connected to server but sometime it like miss and match with this error. Do you know why it happen? This sometime stop and job start, some time it run forever.
I also add this in config_fed_client.json:

  "submit_task_result_timeout":6000,
  "get_task_timeout":6000,
  "max_runner_sync_tries":6000,
  "max_runner_sync_timeout":6000,
  "task_check_timeout":6000,
  "task_check_interval":10,
  "cell_wait_timeout":6000,
  "end_run_readiness_timeout":6000,
  "runner_sync_timeout":2,
  "job_heartbeat_interval":150,
  "end_run_readiness_timeout":100,
  "end_run_readiness_check_interval":2,```



[38m2025-07-11 11:10:43,619 - ClientManager - INFO - Client: New client site-6@10.128.145.232 joined. Sent token: 135a956e-5a8b-4e36-b567-41f2ebe5b238.  Total clients: 16�[0m
�[38m2025-07-11 11:12:02,760 - CoreCell - INFO - Debugging4: The timeout in core_cell is setat: 15�[0m
�[38m2025-07-11 11:12:02,771 - DefaultJobScheduler - INFO - [identity=server, run=?] - Try to schedule job d536dfb2-9cbd-4dfd-ba77-c333e3c930db, get result: (scheduled).�[0m
�[38m2025-07-11 11:12:02,816 - JobRunner - INFO - [identity=server, run=?] - Got the job: d536dfb2-9cbd-4dfd-ba77-c333e3c930db from the scheduler to run�[0m
�[38m2025-07-11 11:12:02,831 - JobRunner - INFO - [identity=server, run=?] - Application app deployed to the server for job: d536dfb2-9cbd-4dfd-ba77-c333e3c930db�[0m
�[38m2025-07-11 11:12:02,831 - JobRunner - INFO - [identity=server, run=?] - App app to be deployed to the clients: site-10,site-15,site-11,site-14,site-9,site-12,site-8,site-16,site-13,site-4,site-3,site-7,site-1,site-5,site-2,site-6 for run: d536dfb2-9cbd-4dfd-ba77-c333e3c930db�[0m
�[38m2025-07-11 11:12:02,832 - CoreCell - INFO - Debugging4: The timeout in core_cell is setat: 10.0�[0m
�[38m2025-07-11 11:12:02,855 - JobRunner - INFO - [identity=server, run=?] - Updated the schedule history of Job: d536dfb2-9cbd-4dfd-ba77-c333e3c930db�[0m
�[38m2025-07-11 11:12:02,863 - ServerProcessJobLauncher - INFO - Launch the job in process ID: 1903526�[0m
�[38m2025-07-11 11:12:02,864 - ServerEngine - INFO - Launch job_id: d536dfb2-9cbd-4dfd-ba77-c333e3c930db  with job launcher: <class 'nvflare.app_common.job_launcher.server_process_launcher.ServerProcessJobLauncher'> �[0m
�[38m2025-07-11 11:12:02,867 - CoreCell - INFO - Debugging4: The timeout in core_cell is setat: 20�[0m
�[38m2025-07-11 11:12:02,882 - JobRunner - INFO - [identity=server, run=?] - Started run: d536dfb2-9cbd-4dfd-ba77-c333e3c930db for clients: site-10,site-15,site-11,site-14,site-9,site-12,site-8,site-16,site-13,site-4,site-3,site-7,site-1,site-5,site-2,site-6�[0m
�[38m2025-07-11 11:12:02,883 - JobRunner - INFO - [identity=server, run=?] - Job: d536dfb2-9cbd-4dfd-ba77-c333e3c930db started to run, status changed to RUNNING.�[0m
�[38m2025-07-11 11:12:04,179 - runner_process - INFO - Runner_process started.�[0m
�[38m2025-07-11 11:12:04,208 - ServerDeployer - INFO - server heartbeat timeout set to 600�[0m
�[38m2025-07-11 11:12:04,384 - CoreCell - INFO - server.d536dfb2-9cbd-4dfd-ba77-c333e3c930db: created backbone internal connector to tcp://localhost:18343 on parent�[0m
�[38m2025-07-11 11:12:04,385 - conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:18343] is starting�[0m
�[38m2025-07-11 11:12:04,395 - conn_manager - INFO - Connection [CN00002 127.0.0.1:44632 => 127.0.0.1:18343] is created: PID: 1903526�[0m
�[38m2025-07-11 11:12:04,396 - Cell - INFO - Register blob CB for channel='server_command', topic='*'�[0m
�[38m2025-07-11 11:12:04,396 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'�[0m
�[38m2025-07-11 11:12:04,397 - ServerCommandAgent - INFO - ServerCommandAgent cell register_request_cb: server.d536dfb2-9cbd-4dfd-ba77-c333e3c930db�[0m
�[38m2025-07-11 11:12:04,397 - runner_process - INFO - Network_monitor  started.�[0m
�[38m2025-07-11 11:12:04,397 - conn_manager - INFO - Connection [CN00018 127.0.0.1:18343 <= 127.0.0.1:44632] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,265 - conn_manager - INFO - Connection [CN00019 0.0.0.0:8002 <= ipv4:10.128.145.232:46884 SSL site-3] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,268 - conn_manager - INFO - Connection [CN00020 0.0.0.0:8002 <= ipv4:10.128.145.232:46902 SSL site-5] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,273 - conn_manager - INFO - Connection [CN00021 0.0.0.0:8002 <= ipv4:10.128.145.232:46876 SSL site-2] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,274 - conn_manager - INFO - Connection [CN00022 0.0.0.0:8002 <= ipv4:10.128.145.232:46916 SSL site-4] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,277 - conn_manager - INFO - Connection [CN00023 0.0.0.0:8002 <= ipv4:10.128.145.232:46932 SSL site-7] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,279 - conn_manager - INFO - Connection [CN00024 0.0.0.0:8002 <= ipv4:10.128.145.232:46886 SSL site-8] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,287 - conn_manager - INFO - Connection [CN00025 0.0.0.0:8002 <= ipv4:10.128.145.232:46940 SSL site-6] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:04,322 - conn_manager - INFO - Connection [CN00026 0.0.0.0:8002 <= ipv4:10.128.145.232:46942 SSL site-1] is created: PID: 1903290�[0m
�[31m2025-07-11 11:13:04,973 - byte_receiver - ERROR - Rx[SID:1753803988816604 from site-5.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,975 - byte_receiver - ERROR - Rx[SID:1753803987816560 from site-2.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,976 - byte_receiver - ERROR - Rx[SID:1753803985816648 from site-6.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,978 - byte_receiver - ERROR - Rx[SID:1753803989816600 from site-7.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,979 - byte_receiver - ERROR - Rx[SID:1753803990816562 from site-3.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,984 - byte_receiver - ERROR - Rx[SID:1753803986816595 from site-8.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,985 - byte_receiver - ERROR - Rx[SID:1753803984816579 from site-4.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:04,987 - byte_receiver - ERROR - Rx[SID:1753803991852641 from site-1.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[38m2025-07-11 11:13:08,386 - conn_manager - INFO - Connection [CN00027 0.0.0.0:8002 <= ipv4:10.128.145.237:58936 SSL site-11] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,390 - conn_manager - INFO - Connection [CN00028 0.0.0.0:8002 <= ipv4:10.128.145.237:58970 SSL site-14] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,393 - conn_manager - INFO - Connection [CN00029 0.0.0.0:8002 <= ipv4:10.128.145.237:58980 SSL site-13] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,398 - conn_manager - INFO - Connection [CN00030 0.0.0.0:8002 <= ipv4:10.128.145.237:58968 SSL site-12] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,399 - conn_manager - INFO - Connection [CN00031 0.0.0.0:8002 <= ipv4:10.128.145.237:58940 SSL site-10] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,402 - conn_manager - INFO - Connection [CN00032 0.0.0.0:8002 <= ipv4:10.128.145.237:58952 SSL site-16] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,403 - conn_manager - INFO - Connection [CN00033 0.0.0.0:8002 <= ipv4:10.128.145.237:58978 SSL site-15] is created: PID: 1903290�[0m
�[38m2025-07-11 11:13:08,441 - conn_manager - INFO - Connection [CN00034 0.0.0.0:8002 <= ipv4:10.128.145.237:58988 SSL site-9] is created: PID: 1903290�[0m
�[31m2025-07-11 11:13:08,754 - byte_receiver - ERROR - Rx[SID:1753803984816580 from site-4.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,755 - byte_receiver - ERROR - Rx[SID:1753803989816601 from site-7.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,756 - byte_receiver - ERROR - Rx[SID:1753803985816649 from site-6.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,756 - byte_receiver - ERROR - Rx[SID:1753803988816605 from site-5.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,757 - byte_receiver - ERROR - Rx[SID:1753803990816563 from site-3.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,757 - byte_receiver - ERROR - Rx[SID:1753803987816561 from site-2.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,768 - byte_receiver - ERROR - Rx[SID:1753803986816596 from site-8.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:08,769 - byte_receiver - ERROR - Rx[SID:1753803991852642 from site-1.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,739 - byte_receiver - ERROR - Rx[SID:1753715357833565 from site-11.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,739 - byte_receiver - ERROR - Rx[SID:1753715356833545 from site-15.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
Some weights of BertForTokenClassification were not initialized from the model checkpoint at /lustre/orion/csc666/proj-shared/oliko/scalable/model/bert-cache and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
�[31m2025-07-11 11:13:09,775 - byte_receiver - ERROR - Rx[SID:1753715355833570 from site-14.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,776 - byte_receiver - ERROR - Rx[SID:1753715361833566 from site-16.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,777 - byte_receiver - ERROR - Rx[SID:1753715358834090 from site-9.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 375] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,777 - byte_receiver - ERROR - Rx[SID:1753715354833554 from site-10.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,778 - byte_receiver - ERROR - Rx[SID:1753715359833566 from site-12.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m
�[31m2025-07-11 11:13:09,779 - byte_receiver - ERROR - Rx[SID:1753715360833576 from site-13.d536dfb2-9cbd-4dfd-ba77-c333e3c930db for aux_communication/__sync_runner__ Size: 378] callback <bound method BlobHandler.handle_blob_cb of <nvflare.fuel.f3.streaming.blob_streamer.BlobHandler object at 0x7fffe64cc730>> throws exception: no run_manager in Server Engine.�[0m

[chesterxgchen]

feel like a a setup issues.
check some basic informations

1. which FLARE version, python version, dev env.
2. do you have the same versions on client code and server code ?
3. did you do the clean installation ?
4. when do you see these errors ? only in certain jobs or all jobs ? can you successfully run the basic hello-world jobs ?