v0.28.0-beta1

v0.28.0-beta1 (2025-04-23)

Full Changelog

Warning

This is a beta release, as it introduces a new Rest client, to replace the use of GraphQL for communicating with Buildkite. As this is a new API, we may make breaking changes to it as we receive feedback from real-world use. As such, if you do use it, we suggest upgrading to the full release once it is available, as APIs the beta relies on may be removed in the future.

Added

• feat: New APIs used to fetch jobs #562 (@DrJosh9000)

Changed

• Update config defaults + docs #570 (@swaller-bk)
• SUP-3454 - Update PodSpec docs, small fix/cleanup #568 (@petetomasik)
• SUP-3717 - Docs/more updates #567 (@petetomasik)
• Validation for the container image name #565 (@nsuma8989)

Internal

• chore(deps): Bump github.com/buildkite/agent/v3 from 3.95.1 to 3.96.0 #564 (@dependabot[bot])

Agent Changelog

v3.96.0 (2025-04-10)

Full Changelog

Added

• Add pause and resume subcommands #3273 (@DrJosh9000)

Internal

• chore: Use golangci-lint for code checks #3274 (@DrJosh9000)
• FakeAPIServer's PingHandler is passed the *http.Request #3271 (@pda)
• FakeAPIServer handles agent registration: AddRegistration(tok, resp) #3272 (@pda)
• fix: ISE message when json.Marshal fails #3270 (@DrJosh9000)
• agent_worker_test: tests for endpoint switching during register/ping #3269 (@pda)
• Refactor fake API server #3264 (@DrJosh9000)
• AgentWorker has noWaitBetweenPingsForTesting field #3262 (@pda)
• refactor: rename AgentRegisterResponse local vars to reg consistently #3259 (@pda)

Dependencies

• Bump the container-images group across 6 directories with 2 updates #3266 (@dependabot[bot])
• Bump the cloud-providers group across 1 directory with 3 updates #3267 (@dependabot[bot])
• Bump the golang-x group with 4 updates #3265 (@dependabot[bot])
• Bump golang.org/x/net from 0.37.0 to 0.38.0 in the golang-x group #3256 (@dependabot[bot])
• Bump the container-images group across 4 directories with 1 update #3258 (@dependabot[bot])
• Bump the cloud-providers group across 1 directory with 2 updates #3252 (@dependabot[bot])
• Bump the container-images group across 5 directories with 2 updates #3251 (@dependabot[bot])
• Bump gopkg.in/DataDog/dd-trace-go.v1 from 1.72.1 to 1.72.2 #3250 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.28.0-beta1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.28.0-beta1
Digest: sha256:4133f65647da1335ca002770251c0fef31243a8fd58b607b105cbb9d2a4e285d

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.28.0-beta1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.28.0-beta1
Digest: sha256:0501ef6ebb873187c9d72b070c1673b07be6a97e7082d63c45c561c045b6f428

Agent

Image: ghcr.io/buildkite/agent:3.96.0
Digest: sha256:31ef093a869468d7c781b86895058773c10cfb646856548c8841822da262864f

v0.27.0

v0.27.0 (2025-04-14)

Full Changelog

Fixed

• SUP-3616 - Fix duplicate calculation/application of podSpecPatch #555 (@petetomasik)
• Fix the max-in-flight default value in schema #559 (@nsuma8989)
• Skip jobs that are missing queue tag, enhance info logs #560 (@petetomasik)

Internal

• Reworking the Docs section #536 (@karensawrey)
• ✨ Improve issue raising, add templates #550 (@mcncl)
• fix: migrate golangci-lint to v2 #556 (@scadu)

Dependencies

• chore(deps): Bump gotest.tools/gotestsum from 1.12.0 to 1.12.1 #540 (@dependabot[bot])
• chore(deps): Bump github.com/buildkite/agent/v3 from 3.93.1 to 3.95.1 #551 (@dependabot[bot])
• chore(deps): Bump the k8s group across 1 directory with 4 updates #552 (@dependabot[bot])
• chore(deps): Bump github.com/spf13/viper from 1.19.0 to 1.20.1 #557 (@dependabot[bot])
• chore(deps): Bump github.com/go-playground/validator/v10 from 10.25.0 to 10.26.0 #558 (@dependabot[bot])
• chore(deps): Bump github.com/prometheus/client_golang from 1.21.1 to 1.22.0 #563 (@dependabot[bot])

New Contributors

• @mcncl made their first contribution in #550

Agent Changelog

v3.95.1 (2025-03-20)

Full Changelog

[!IMPORTANT]
Secrets (as visible to the agent in environment variables) are now redacted from annotations, meta-data values, and step updates, similar to how secrets are redacted from job logs.
If needed, this can be disabled by passing the flag --redacted-vars='' to the annotate, meta-data set, or step update command.

Security

• Fix incomplete processing in newly-redacted operations #3246 (@DrJosh9000)
• Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 (resolves CVE-2025-30204) #3247 (@dependabot[bot])

v3.95.0 (2025-03-20)

Full Changelog

[!IMPORTANT]
Secrets (as visible to the agent in environment variables) are now redacted from annotations, meta-data values, and step updates, similar to how secrets are redacted from job logs.
If needed, this can be disabled by passing the flag --redacted-vars='' to the annotate, meta-data set, or step update command.

Changed

• Redact meta-data values and step attribute updates with warnings #3243 (@DrJosh9000)
• Redact annotations #3242 (@DrJosh9000)
• ANSI parser speedup #3237 (@DrJosh9000)

Fixed

• Agents running with disconnect-after-job or disconnect-after-idle-timeout can now be kept alive with agent pausing #3238 (@DrJosh9000)
• The pty-raw experiment no longer causes a warning to be logged #3241 (@DrJosh9000)

Dependency updates

• Bump google.golang.org/api from 0.224.0 to 0.226.0 in the cloud-providers group #3240 (@dependabot[bot])
• Bump the container-images group across 7 directories with 3 updates #3239 (@dependabot[bot])

v3.94.0 (2025-03-12)

Full Changelog

Security

• Fix redaction of multiline secrets printed as single lines #3233 (@DrJosh9000)

Changed

• Change healthHandler to log requests at debug level #3232 (@DrJosh9000)
• go.mod: go 1.23.0, toolchain go1.23.7 #3225 (@DrJosh9000)
• Record build URL in the buildkite-agent log for easier traceability #3215 (@mkrapivner-zipline)

Added

• Adding an initial bazel configuration #3141 (@CerealBoy)

Dependency bumps

• #3228, #3230, #3229, #3231, #3222, #3220, #3221, #3216, #3217, #3218 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.27.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.27.0
Digest: sha256:6a27bf70d58d64e2d76845f68ef4ce5e5a6b42fdc8d7f0bb793251e5d38fb89a

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.27.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.27.0
Digest: sha256:c101264f367383e33b7937b204de3ccaf8f19784c37d8df33eb007c55ee5885c

Agent

Image: ghcr.io/buildkite/agent:3.95.1
Digest: sha256:7e813c353bd315af56165c84837d221704ede175c9e0f715260df33ceb040231

v0.26.3

v0.26.3 (2025-03-21)

Full Changelog

Internal

• chore: add additional logging and metrics for limiter #549 (@scadu)

New Contributors

• @scadu made their first contribution in #549

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.26.3
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.26.3
Digest: sha256:83b5e818262b3ba03a1465b78a77c547c5c78bfb01ad1d00f41ea4357ce47133

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.26.3
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.26.3
Digest: sha256:58471d09c275034fd18d7272d258cb3ffc17649149ad70ee35536b292e84b092

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.26.2

v0.26.2 (2025-03-18)

Full Changelog

Fixed

• SUP-3556: De-dupe VolumeMounts in generated PodSpec #547 (@petetomasik)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.26.2
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.26.2
Digest: sha256:53565f98cad651f254995a0a44feeff7469c24d20796f8872fd09b1bea88a527

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.26.2
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.26.2
Digest: sha256:75defe7b23bad121719f5be04a1ec753a4610fcd119f1d6904051c08b9f8bfea

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.26.1

v0.26.1 (2025-03-17)

Full Changelog

Fixed

• Dedupe extraVolumeMounts for checkout, command, sidecar params #545 (@petetomasik)

Dependencies

• chore(deps): Bump github.com/prometheus/client_golang from 1.21.0 to 1.21.1 #538 (@dependabot[bot])
• chore(deps): Bump sigs.k8s.io/controller-runtime from 0.20.2 to 0.20.3 in the k8s group #537 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.26.1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.26.1
Digest: sha256:54fa61774e9651c49d84ffa154d4122b9a0dc5abe5a036168335025ec9c85a51

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.26.1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.26.1
Digest: sha256:040abd927d34dd5dc5aec8e3202b41661f608306f5a44e5197f15400e8ccd7ff

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.26.0

v0.26.0 (2025-03-10)

Full Changelog

Added

• SUP-3228: Allow extraVolumeMounts for default params #534 (@petetomasik)
• Allow configurable job name prefix #526 (@petetomasik)
• SUP-3316: Add debug logging for stale jobs due to max-in-flight capacity #533 (@petetomasik)

Fixed

• SUP-3496: Ensure token is returned to bucket when BK jobs are acquired and failed #535 (@petetomasik)

Dependencies

• chore(deps): Bump github.com/jedib0t/go-pretty/v6 from 6.6.6 to 6.6.7 #532 (@dependabot[bot])

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.26.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.26.0
Digest: sha256:f42cba527290df88771f6ec7221c92df915d5ffed0c85b498a85d6659c2e60e6

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.26.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.26.0
Digest: sha256:a190cb7675001108ce9c77c2f4b13ab4af0d00f935a97fac6bf8421cef781d9e

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.25.0

v0.25.0 (2025-02-28)

Full Changelog

Added

• SUP-3347: Enable Pagination of Scheduled Jobs #515 (@tomowatt)

Internal

• Adds the information on cluster's UUID in a structured form + minor improvements #531 (@karensawrey)

Dependencies

• Bump buildkite-agent to v3.93.1 #529 (@DrJosh9000)

Agent Changelog

v3.93.1 (2025-02-27)

Full Changelog

Added

• Set env when job cancelled for hooks #3213 (@sj26)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.25.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.25.0
Digest: sha256:f897db57354f239fdd7a3f67c7ae8aef56b33af3587529432ccd8bfd5e6dbad4

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.25.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.25.0
Digest: sha256:bf1e138c19318522c5db62b0f62f68fc900194cd9b97ae0017cfab3caa48e51e

Agent

Image: ghcr.io/buildkite/agent:3.93.1
Digest: sha256:03a6fb7b4d98bbf479e687317b42886d2e2a51e0296b669b9fa53c7cca5cc0fb

v0.24.1

v0.24.1 (2025-02-26)

Full Changelog

Fixed

• Fix labels for pod eviction metric by @petetomasik in #525
• Check that CheckoutParams.GitCredentialsSecret is not nil by @petetomasik in #527

Dependencies

• Bump github.com/prometheus/client_golang from 1.20.5 to 1.21.0 by @dependabot in #524
• Bump github.com/google/go-cmp from 0.6.0 to 0.7.0 by @dependabot in #523

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.24.1
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.24.1
Digest: sha256:1f888b75050232efa0f173692ba37c018e1d3ed2b83f93876887b202d9d9ae5a

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.24.1
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.24.1
Digest: sha256:d6cea6214d9d8b5d8b4d4509eb06095d0a62841bd2889db1804eb114ef4820e6

Agent

Image: ghcr.io/buildkite/agent:3.92.1
Digest: sha256:2a1be0f1907b4117bd91f3c89ed849248355adcff9f17f52be3b6693c173c97f

v0.24.0

v0.24.0 (2025-02-24)

Full Changelog

Note

The controller will now add .spec.activeDeadlineSeconds to every Kubernetes Job it creates. This value defaults to 21600 seconds (6 hours) and can be overwritten directly in the controller's configuration or on a per-job basis with the kubernetes plugin. This active deadline will ensure that no Kubernetes Job is allowed to run indefinitely.

Added

• SUP-3258 - Implement .spec.job.activeDeadlineSeconds by @petetomasik in #497
• [SUP-3361] Controller to handle queue paused events and prevent processing jobs by @nsuma8989 in #507
• Annotate pods with job priority number by @mikehw in #519
• Add config option to disable command modification check by @DrJosh9000 in #449

Fixed

• Fix unusable image pull policy options by @DrJosh9000 in #508

Changed

• Account for reason DeadlineExceeded in checkFinishedWithoutPod() by @petetomasik in #517

Internal

• Update CODEOWNERS by @karensawrey in #514
• [SUP-3361] Update readme with information about queue pause config by @nsuma8989 in #521

Dependencies

• Bump the k8s group with 4 updates by @dependabot in #509
• Bump github.com/spf13/pflag from 1.0.5 to 1.0.6 by @dependabot in #493
• Bump github.com/Khan/genqlient from 0.7.0 to 0.8.0 by @petetomasik in #516
• Bump github.com/buildkite/agent/v3 from 3.91.0 to 3.92.1 by @dependabot in #510
• Bump github.com/jedib0t/go-pretty/v6 from 6.6.5 to 6.6.6 by @dependabot in #511
• Bump github.com/go-playground/validator/v10 from 10.24.0 to 10.25.0 by @dependabot in #512
• Bump github.com/spf13/cobra from 1.8.1 to 1.9.1 by @dependabot in #513

New Contributors

• @mikehw made their first contribution in #519

Agent Changelog

v3.92.1 (2025-02-13)

Full Changelog

Removed

• Revert "Ensure the log streamer respects forced shutdown of the agent" #3191 (@wolfeidau)
• Revert "Fix data race on exitImmediately" #3190 (@wolfeidau)

Dependabot

• The usual updates: #3188, #3185 (@dependabot[bot])

[!NOTE]
Reverted #3180 and #3187 as this change introduced a bug which resulted in truncated log output. Will re-think this fix and push it out again in another release after we do some more testing.

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.24.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.24.0
Digest: sha256:ed9257b4454d7d9278ea70955a0dce0b5aacf9915af310d3f3a4a7a3646feec4

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.24.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.24.0
Digest: sha256:1aadf37418c702e5f4755594bb758a6c3613ed5f382aac5d7ebfc1df6c5a5a3a

Agent

Image: ghcr.io/buildkite/agent:3.92.1
Digest: sha256:2a1be0f1907b4117bd91f3c89ed849248355adcff9f17f52be3b6693c173c97f

v0.23.0

v0.23.0 (2025-02-12)

Full Changelog

Added

• SUP-3261: Use Protobuf encoding for API responses by @petetomasik in #501
• Allow configurable QPS, Burst settings with Kubernetes client, increase defaults by @petetomasik in #489
• Add note about DSA keys deprecation, change example to RSA by @petetomasik in #471
• SUP-3283: Enable configuring GraphQL Result Limits by @tomowatt in #499
• enable name overrides by @alexr56k in #490
• SUP-3349: Apply sane resource requests to imagecheck containers by @petetomasik in #504

Fixed

• Fix resources and port parsing for non-strings by @mikekap in #495

Changed

• Update CODEOWNERS - adding Support Engineers as CODEOWNERS by @karensawrey in #494
• Update CODEOWNERS - remove obsolete team by @karensawrey in #496
• SUP-3216 - Remove checkout container from podSpec, if skipped by @petetomasik in #502

Dependencies

• Bump sigs.k8s.io/controller-runtime from 0.20.0 to 0.20.1 in the k8s group by @dependabot in #485
• Bump github.com/buildkite/agent/v3 from 3.90.0 to 3.91.0 by @dependabot in #492

New Contributors

• @petetomasik made their first contribution in #471
• @karensawrey made their first contribution in #494
• @mikekap made their first contribution in #495
• @tomowatt made their first contribution in #499
• @alexr56k made their first contribution in #490

Agent Changelog

v3.91.0 (2025-01-28)

Full Changelog

Changed

• Jitter within ping, status, log loops #3164 (@DrJosh9000)

Fixed

• Roko v1.3.1 #3157 (@moskyb)
• Better plugin checkout logging #3166 (@DrJosh9000)

Internal

• Add /.buildkite dir for Dockerfile updates #3162 (@DrJosh9000)

Images

Helm chart

Image: public.ecr.aws/buildkite/helm/agent-stack-k8s:0.23.0
Image: ghcr.io/buildkite/helm/agent-stack-k8s:0.23.0
Digest: sha256:d6194df0439771a929b8bf133a1689a4f956a32a9b0b369694a516de4230e876

Controller

Image: public.ecr.aws/buildkite/agent-stack-k8s/controller:0.23.0
Image: ghcr.io/buildkite/agent-stack-k8s/controller:0.23.0
Digest: sha256:6e5c269e1925b2ec0db2aaa5f41ea1b5a28ed8167b60be1f900d54d8c327579e

Agent

Image: ghcr.io/buildkite/agent:3.91.0
Digest: sha256:f46a442f89dec73a979d9ecfd0a7287dde82fefe0e3603aeff3c8324e98373ca