protobuf runtime issue: google.protobuf.runtime_version.VersionError

[jieyu-lin]

What's the issue?

When running new build code location with dagster version 1.12.22 in kubernetes, it raises error with following:

protobuf runtime issue: google.protobuf.runtime_version.VersionError. Detected incompatible Protobuf Gencode/Runtime versions when loading grpc_health/v1/health.pr. gencode 7.35.0 runtime 6.33.6. Runtime version cannot be older than the linked gencode version. See Protobuf version guarantees at https://protobuf.dev/support/cross-version-runtime-guarantee.

Then, all code locations remain in state CrashBackoff all the time, and keep restarting.

What did you expect to happen?

dependencies = [
    "dagster>=1.12,<1.13",
    "dagster-webserver>=1.12,<1.13",
    "dagster-postgres>=0.28,<0.29",
    "dagster-aws>=0.28,<0.29",
    "dagster-graphql>=1.12,<1.13",
    "dagster-slack>=0.28,<0.29",
    "dagster-duckdb>=0.28,<0.29",
]

When uv sync those dependencies, we can build image accordingly. But when runtime, it start failure. Based on code base review, I can get following conclusion:

1. protobuf==6.33.6 satisfies Dagster’s protobuf<7
2. grpcio-health-checking==1.82.0 is allowed by Dagster’s >=1.44.0 floor
3. But the 1.82 wheel embeds gencode 7.35.0 while runtime is 6.33.6 → exact error

Dagster imports grpc_health at code-server startup and pins protobuf <7 but does not cap grpcio-health-checking / grpcio. After grpc 1.82.0 (2026-07-06), fresh installs can resolve a broken combo.

This issue can be simply resolved by adding constraint-dependencies like:

[tool.uv]
constraint-dependencies = [
    "grpcio-health-checking<1.82",
    "grpcio<1.82",
]

Or directly add to dependencies:

dependencies = [
    "dagster>=1.12,<1.13",
    "dagster-webserver>=1.12,<1.13",
    "dagster-postgres>=0.28,<0.29",
    "dagster-aws>=0.28,<0.29",
    "dagster-graphql>=1.12,<1.13",
    "dagster-slack>=0.28,<0.29",
    "dagster-duckdb>=0.28,<0.29",
    "grpcio-health-checking<1.82",
    "grpcio<1.82",
]

But still require caps added into code base to prevent inconsistency.

How to reproduce?

Try to spin up code location with following example pyproject.toml dependencies and start in kubernetes with 1.12.22 version

[project]
dependencies = [
    "dagster>=1.12,<1.13",
    "dagster-webserver>=1.12,<1.13",
    "dagster-postgres>=0.28,<0.29",
    "dagster-aws>=0.28,<0.29",
    "dagster-graphql>=1.12,<1.13",
    "dagster-slack>=0.28,<0.29",
    "dagster-duckdb>=0.28,<0.29",
]

Dagster version

1.12.22

Deployment type

Dagster Helm chart

Deployment details

Simply build docker image by python:3.12-slim, and install pyproject.toml through uv venv and uv sync --refresh. After that, we spin up a dagster/dagster-celery-k8s with version 1.12.22 to connect this code location via new build image, simulate in my kubernetes using rancher-desktop

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[ahmedhamidibr]

Confirming the same error on a Docker (non-k8s) deployment:

dagster==1.12.14
dagster-webserver==1.12.14

Pinning in the requirements of my code location / workspace images fixed it:

grpcio-health-checking==1.80.0

1.80.0 appears to be the last release built with protobuf 6.x gencode, consistent with the <1.82 constraint suggested above. +1 for adding an upper cap in Dagster itself so fresh builds can't resolve the broken combo.

[trexx]

grpcio-health-checking 1.81.1 ships gencode 6.33.5 and as of posting is the latest version compatible.

[0xliam]

I am new to Dagster, and ran into this today. I opened a merge request to cap grpcio/grpcio-health-checking - feedback is appreciated! #33979

[jieyu-lin]

I try the same using uv by adding constraint to pyproject.toml I mentioned, pin version below 1.82 can have temporarily fix for dagster 1.12

[tool.uv]
constraint-dependencies = [
    "grpcio-health-checking<1.82",
    "grpcio<1.82",
]

Any new request changes may merge directly to dagster 1.13 unless dagster team backfill this to previous

[miclindahl]

Are there any solutions/workaround for this? I haven't been able to deploy since yesterday. I'm on Dagster plus.

[CommonCrisis]

Same here I cannot deploy Dagster on AWS ECS at all.

[0xliam]

Looks like this is resolved by this commit - a249c55

[Aluriak]

Thank you everyone. On our deployment, we simply ran pip install -U 'protobuf<7,>=4' 'grpcio-health-checking<1.82' 'grpcio<1.82' to get it working. To avoid problem in the future, we added those constraints in the pyproject.toml dependencies.