ci(buildkite): migrate dagster-aws, dagster-dask, dagster-azure-live-tests to EKS (#24716)

## Summary & Motivation

Follow-up to #24661. Drops three more OSS test suites off the MEDIUM
(EC2) queue onto `kubernetes-eks`.

The three suites were on MEDIUM with `env_vars=[AWS_*]` or
`env_vars=[TEST_AZURE_*]` configured, which made it look like a
secrets-provisioning blocker. On inspection it isn't:

| Suite | Reality |
|---|---|
| **`dagster-aws`** | `conftest.py` autouse fixture sets fake creds with
`monkeypatch.setenv("AWS_ACCESS_KEY_ID", "test")` etc., tests use
`moto.mock_aws` — no real AWS access. The `env_vars=[AWS_*]` was
vestigial. |
| **`dagster-dask`** | Tests don't import boto3 or touch AWS at all. The
`env_vars=[AWS_*]` was fully vestigial. |
| **`dagster-azure-live-tests`** | `TEST_AZURE_*` keys already live in
the EKS K8s secret `buildkite-dagster-secrets`
(`infra/k8s/buildkite/overlays/buildkite-eks/secrets.yaml:62-67`), which
is auto-mounted on every step pod via `envFrom`
(`command_step_builder.py:582-589`). So the suite's
`env_vars=[TEST_AZURE_*]` just allowlists the names through the
docker→pod boundary, and the actual values reach the test process for
free. |

So there's no secret-provisioning work to do — just dropping the
`queue=BuildkiteQueue.MEDIUM` toggle and cleaning up the dead AWS
`env_vars` lists on `dagster-aws` / `dagster-dask`.

## Changes

-
`dagster-oss/.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py`
- `dagster-aws`: drop `queue=BuildkiteQueue.MEDIUM` and
`env_vars=[AWS_*]`. PackageSpec is now bare.
  - `dagster-dask`: same.
-
`dagster-oss/.buildkite/dagster-buildkite/dagster_buildkite/steps/integration.py`
- `dagster-azure-live-tests`: drop `queue=BuildkiteQueue.MEDIUM`. Keep
the `env_vars=[TEST_AZURE_*]` allowlist (still needed at the docker
boundary).

Starting with default EKS sizing on all three (outer pod 1000m cpu / no
memory request → BestEffort QoS, dind sidecar 2000m / 1Gi req / 2Gi
limit / 10Gi ephemeral storage). The Azure suite runs `just rebuild_ui`
plus a `dagster-webserver` subprocess, which could be RAM-hungry;
flagging that as the likely candidate for a follow-up bump if it OOMs.

## Test Plan

- [ ] CI passes on this PR.
- [ ] On the resulting build, all three step keys (`dagster-aws-*`,
`dagster-dask-*`, `dagster-azure-live-tests-*`) report `agent:
kubernetes-eks` rather than `medium`.
- [ ] The Azure suite isn't selectable from this PR (it has
`skip_if_not_azure_commit`); follow-up: ensure the next
dagster-azure-touching PR exercises it on EKS.
- [ ] Watch the first ~5–10 master builds post-merge. If
`dagster-azure-live-tests` OOMs on the `yarn rebuild_ui` step or the
dagster-webserver subprocess, follow up with
`ResourceRequests(cpu="1000m", memory="1Gi", docker_memory="2Gi",
docker_memory_limit="4Gi")` — same shape as the dagster-mysql bump in
#24661.

## Remaining OSS MEDIUM-queue surface

After this lands, the remaining `queue=BuildkiteQueue.MEDIUM` entries in
the OSS step builders are:

- Test-project-blocked (waiting on the ECR repo policy in #24713 to
apply): `test-project-image` build, `dagster-celery-docker`,
`dagster-docker`
- Airflow / airlift-bound: `dagster-airlift`,
`dagster-airlift/perf-harness`, `dagster-test`
- Compose / multi-container fixtures: `backcompat-test-suite`,
`daemon-test-suite`
- Unexplained: `dagster-graphql`
- Per-factor outliers: `dagster-dg-cli` `slow`, `dagstermill`
`papermill2`

## Changelog

NOCHANGELOG

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Internal-RevId: b557ef90abc6ef78190f1e730f8bee95aa4a5856

Author: 2 people

.buildkite/dagster-buildkite/dagster_buildkite/steps/integration.py

@@ -171,7 +171,6 @@ def build_azure_live_test_suite_steps(ctx: BuildkiteContext) -> list[TopLevelSte
             "TEST_AZURE_CONTAINER_ID",
             "TEST_AZURE_ACCESS_KEY",
         ],
-        queue=BuildkiteQueue.MEDIUM,
     ).build_steps(ctx)
 
 

.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py

@@ -1118,8 +1118,6 @@ def _library_packages_with_custom_config(ctx: BuildkiteContext) -> list[PackageS
         ),
         PackageSpec(
             oss_path("python_modules/libraries/dagster-aws"),
-            env_vars=["AWS_DEFAULT_REGION", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"],
-            queue=BuildkiteQueue.MEDIUM,
         ),
         PackageSpec(
             oss_path("python_modules/libraries/dagster-azure"),
@@ -1139,8 +1137,6 @@ def _library_packages_with_custom_config(ctx: BuildkiteContext) -> list[PackageS
         ),
         PackageSpec(
             oss_path("python_modules/libraries/dagster-dask"),
-            env_vars=["AWS_SECRET_ACCESS_KEY", "AWS_ACCESS_KEY_ID", "AWS_DEFAULT_REGION"],
-            queue=BuildkiteQueue.MEDIUM,
         ),
         PackageSpec(
             oss_path("python_modules/libraries/dagster-databricks"),

python_modules/libraries/dagster-aws/dagster_aws_tests/pipes_tests/fake_glue.py

@@ -1,3 +1,4 @@
+import os
 import sys
 import tempfile
 import time
@@ -114,6 +115,11 @@ def start_job_run(self, JobName: str, Arguments: dict[str, str] | None, **kwargs
         popen = Popen(
             [sys.executable, f.name, *args],
             env={
+                # Pass through AWS_* creds from the parent (set by the autouse
+                # `fake_aws_credentials` fixture to stub values). Without this,
+                # botocore inside the subprocess has no creds at all and only
+                # works incidentally when the host has an EC2 IMDS instance role.
+                **{k: v for k, v in os.environ.items() if k.startswith("AWS_")},
                 "AWS_ENDPOINT_URL": self.aws_endpoint_url,
                 "TESTING_PIPES_MESSAGES_BACKEND": self.pipes_messages_backend,
             },

python_modules/libraries/dagster-aws/dagster_aws_tests/pipes_tests/fake_lambda.py

@@ -86,7 +86,12 @@ def invoke(self, **kwargs):
                         out_path,
                     ],
                     check=False,
-                    env={},  # env vars part of lambda fn definition, can't vary at runtime
+                    # Real lambda env is fixed at fn definition, but botocore
+                    # inside the subprocess still needs AWS_* creds to sign
+                    # requests (even against moto). Pass through stub creds
+                    # set by the autouse `fake_aws_credentials` fixture; without
+                    # this we only got creds via EC2 IMDS on the MEDIUM queue.
+                    env={k: v for k, v in os.environ.items() if k.startswith("AWS_")},
                     stdout=log_file,
                     stderr=log_file,
                 )