[dagster-celery-docker] environment values containing '=' crash in container_kwargs

[mattfaltyn]

What's the issue?

dagster-celery-docker crashes when celery_docker_executor receives Docker SDK-style list-form environment variables whose values contain =.

The relevant branch in python_modules/libraries/dagster-celery-docker/dagster_celery_docker/executor.py handles docker.container_kwargs.environment when it is not a dict:

for v in e_vars:
    key, val = v.split("=")
    docker_env[key] = val

That fails for valid environment values such as TOKEN=a=b with:

ValueError: too many values to unpack (expected 2)

This also appears inconsistent with Dagster's existing parse_env_var() helper, which uses split("=", maxsplit=1) and already has a regression test for values containing additional equals signs.

What did you expect to happen?

List-form environment values should preserve everything after the first equals sign.

For example, TOKEN=a=b should be passed to Docker as:

{"TOKEN": "a=b"}

How to reproduce?

From the Dagster repo root, this minimal script exercises the shipped create_docker_task path while faking the Dagster instance and Docker client boundaries, so no Docker daemon is required:

uv run --frozen --project python_modules/libraries/dagster-celery-docker python - <<'PY'
from types import SimpleNamespace
import dagster_celery_docker.executor as ex

ex.unpack_value = lambda packed, as_type: SimpleNamespace(
    instance_ref="ref",
    run_id="run-id",
    step_keys_to_execute=["step"],
    get_command_args=lambda: ["dagster", "api", "execute_step"],
)
ex.serialize_value = lambda value: value
ex.DagsterInstance = SimpleNamespace(
    from_ref=lambda ref: SimpleNamespace(
        get_run_by_id=lambda run_id: SimpleNamespace(),
        report_engine_event=lambda *args, **kwargs: "engine-event",
    )
)
ex.docker.client.from_env = lambda: SimpleNamespace(
    containers=SimpleNamespace(run=lambda *args, **kwargs: b"")
)

class FakeCeleryApp:
    def task(self, **_kwargs):
        return lambda fn: fn

task = ex.create_docker_task(FakeCeleryApp())
task(
    SimpleNamespace(request=SimpleNamespace(hostname="worker")),
    {},
    {"image": "busybox", "container_kwargs": {"environment": ["TOKEN=a=b"]}},
)
PY

Actual result:

ValueError: too many values to unpack (expected 2)

The same issue would be hit by executor config shaped like:

execution:
  config:
    docker:
      image: busybox
      container_kwargs:
        environment:
          - TOKEN=a=b

Dagster version

dagster, version 1!0+dev

Observed on current master at commit 93210ecd45b5c0ca6d68477d28bcafd22d9ed233.

Deployment type

Other Docker-based deployment

Deployment details

Affected surface is dagster-celery-docker with celery_docker_executor and docker.container_kwargs.environment supplied as a list of KEY=VALUE strings.

Additional information

A small fix should be to reuse the existing dagster._core.utils.parse_env_var() helper in the list branch instead of splitting manually. Nearby regression coverage could go in python_modules/libraries/dagster-celery-docker/dagster_celery_docker_tests/test_execute_docker.py or a focused unit test for create_docker_task.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[mattfaltyn]

Proposed fix is open in #34007.