dagster and ray clashing for 3.13

[geoHeil]

See ray-project/ray#55466 (comment) for my example dagster and ray are clashing when both are executed on 3.13 python.

The simple 3.12 one works just fine.

What might be causing the problems?

So far, I know this is only a problem for aarch apple silicon chips - it does not occur on x86 linux

[danielgafni]

Hey Georg, I didn't have the time to ensure 3.13 support yet. It isn't supported right now, I am aware of this problem. I am planning to spend some time on this in the near feature, help would be appreciated!

[geoHeil]

I am up for helping. So far I fell a bit puzzled where to start given the the (suspected) clash of the grpc servers.

[danielgafni]

Did you try increasing logging verbosity?

[geoHeil]

So far I could not find this:

2025-08-11 16:34:29 +0200 - dagster-webserver - INFO - Serving dagster-webserver on http://127.0.0.1:3000 in process 51914
2025-08-11 16:34:43 +0200 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - Priority sorting and checking tag concurrency limits for queued runs.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1754922883.969482 5047837 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
I0000 00:00:1754922883.974921 5047837 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
2025-08-11 16:34:44 +0200 - dagster.daemon.QueuedRunCoordinatorDaemon - INFO - Launched 1 runs.

and in the structured event log:

Launching subprocess for "ray_minithing".
Executing step "ray_minithing" in subprocess.
Starting initialization of resources [io_manager, ray_cluster].
Connecting to a local Ray cluster...

On 3.12 an example of working logs looks like ray-project/ray#55466 (comment)

[geoHeil]

Setting

import logging
import os
import sys

# Set environment variables before importing anything else
os.environ.update({
    'GRPC_VERBOSITY': 'DEBUG',
    'GRPC_TRACE': 'all',
    'RAY_LOG_LEVEL': 'DEBUG',
    'RAY_DISABLE_IMPORT_WARNING': '1',
    'PYTHONUNBUFFERED': '1',
    'PYTHONFAULTHANDLER': '1',
})

# Configure logging
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler('debug.log')
    ]
)

results in the following logs https://gist.github.com/geoHeil/6ffd67c28599d4efee80bc2b35c7c853

LLMs sumarize for me (as a non-grpc expert)

    gRPC's C-core library runs background threads for networking, keepalives, etc.
    If the fork() happens while these gRPC threads in the parent process hold locks, the child process inherits the memory with those locks in a locked state.
    However, the child process does not inherit the parent's threads. The thread that was supposed to unlock the lock in the parent doesn't exist in the child.
    When the child process (now running your asset code) tries to initialize Ray, Ray's own gRPC components try to acquire these same (or similar) locks, find them locked, and wait forever. This is a classic deadlock.

However, setting

import os
os.environ["GRPC_FORK_SUPPORT_ENABLED"] = "1"

did also not fix things for me.

Further - the grpc version seems t obe the same accross python versions:

for 3.13

pixi tree -i grpcio -e dev
Environment: dev

grpcio_health_checking 1.71.2 
└── dagster 1.11.5 
    ├── dagster_webserver 1.11.5 
    ├── dagster_graphql 1.11.5 
    │   └── dagster_webserver 1.11.5 (*)
    └── dagster_ray 0.0.22 

grpcio 1.74.0 
├── grpcio_health_checking 1.71.2 (*)
└── dagster 1.11.5 (*)

and for 3.12

pixi tree -i grpcio -e dev
Environment: dev

grpcio 1.74.0 
├── dagster 1.11.5 
│   ├── dagster_graphql 1.11.5 
│   │   └── dagster_webserver 1.11.5 
│   ├── dagster_ray 0.0.22 
│   └── dagster_webserver 1.11.5 (*)
└── grpcio_health_checking 1.71.2 
    └── dagster 1.11.5 (*)

grpcio_health_checking 1.71.2 (*)