Commit 1912dcc
infra/buildkite: drop obsolete pre-checkout/pre-exit hooks + docker_prune (#23386)
## Summary & Motivation
Companion to #23382. Removes both `.buildkite/hooks/pre-checkout` and `dagster-oss/.buildkite/hooks/pre-checkout`, whose only contents were:
```sh
export BUILDKITE_GIT_CLONE_FLAGS="--filter tree:0"
export BUILDKITE_GIT_CLONE_MIRROR_FLAGS="--filter tree:0"
```
These were intended to make buildkite-agent clones partial. After the agent v3.69 → v3.123.1 upgrade (bundled with the agent-stack-k8s 0.10 → 0.42 controller bump), both env vars are honored end-to-end in ways that bake `partialclonefilter=tree:0` into git config and break our SSH-less `container-0`:
- **`BUILDKITE_GIT_CLONE_MIRROR_FLAGS`** is passed when the agent first creates the bare mirror at `/tmp/git-mirror/...` (`internal/job/checkout.go:408`). The flag records `partialclonefilter=tree:0` in the **mirror's** `.git/config`, which is sticky across fetches — the mirror stays partial forever.
- **`BUILDKITE_GIT_CLONE_FLAGS`** is prepended to the working clone command (`internal/job/checkout.go:884-896`), giving an effective command of:
```
git clone --filter tree:0 --reference <mirror> --dissociate <repo-url> .
```
This still records `extensions.partialClone=origin` and `partialclonefilter=tree:0` in the **working clone's** `.git/config`. So even with a full mirror, subsequent fetches in the working tree honor the filter.
Either path causes the master `:pipeline:` job's `_discover_changed_files` step (`git diff HEAD^...HEAD`) to trigger a lazy promisor fetch when the relevant tree isn't materialized. The fetch runs in `container-0`, which has no SSH credentials, and dies with:
```
error: cannot run ssh: No such file or directory
fatal: promisor remote: unable to fork off fetch subprocess
```
#23382 already removed the controller-side enforcement at `gitMirrors.cloneFlags`. The env vars in these hooks were a second mechanism producing the same outcome.
## Also: drop docker_prune + collapse the k8s-gating in pre-command
Now that every Buildkite job runs in EKS, the `docker_prune.sh` script (both copies) is dead code — the script itself short-circuits under `KUBERNETES_PORT` and prints `"don't docker prune, we're running in kubernetes"`. Removing it lets us also drop the `pre-exit` hooks, which existed only to source it.
While here, the `KUBERNETES_PORT`-vs-`/workdir` fork at the top of `.buildkite/hooks/pre-command` collapses to a single unconditional block — the `elif [ -n "${BUILDKITE:-}" ]` branch (assumed non-k8s, used `/workdir` paths) is unreachable now.
Net diff for this second part:
- Delete `.buildkite/scripts/docker_prune.sh` and `dagster-oss/.buildkite/buildkite-shared/buildkite_shared/scripts/docker_prune.sh`.
- Delete both `pre-exit` hooks (each was a single `. ./<docker_prune>` line).
- Drop the `if [ -z "${PYTEST_CURRENT_TEST:-}" ]; then . docker_prune.sh; fi` block from both `pre-command` hooks.
- Replace the `if KUBERNETES_PORT … elif BUILDKITE …` ladder in `.buildkite/hooks/pre-command` with the k8s branch's body, unconditional.
## Test Plan
- Watch master `:pipeline:` jobs on builds that hit freshly autoscaled GKE nodes — they should stop failing with the SSH/promisor error once the mirror is recreated under the new config.
- For nodes that still have a partial mirror on hostPath (`/var/local/dagster-internal/`), `rm -rf` of the mirror subdir is safe (verified against v3.123.1 source: `checkout.go:373-422` — agent stats the mirror dir, recreates via `git clone --mirror` if absent, and self-cleans on partial failure). The triggering job pays an extra ~30s for the recreate but does not fail.
- Verify a representative internal pytest job (e.g. dagster-cloud-dagit) and an OSS-step job both still resolve `DAGSTER_INTERNAL_GIT_REPO_DIR` / `PYTEST_ADDOPTS` correctly after the pre-command collapse — `DAGSTER_INTERNAL_GIT_REPO_DIR=$(pwd)` should land on the workspace root in both cases.
Internal-RevId: 99682f70da1bc33bf66094271242307989a562ca1 parent ec90168 commit 1912dcc
4 files changed
Lines changed: 0 additions & 28 deletions
Lines changed: 0 additions & 21 deletions
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | 3 | | |
8 | 4 | | |
9 | 5 | | |
| |||
This file was deleted.
0 commit comments