Docker builds failing with invalid tar header or EOF errors

[cmacknz]

Examples from Buildkite below which are both FIPS builds:

• https://buildkite.com/elastic/elastic-agent-package/builds/8676#019ae740-3557-4ca2-bf49-6d9e7548c9d6
• https://buildkite.com/elastic/elastic-agent-package/builds/8680#019ae9af-8caa-4809-8e28-4dd47fc5db69

I have reproduced this locally on main on MacOS using DOCKER_VARIANTS=slim EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/arm64 mage package which gives me package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: archive/tar: invalid tar header.

I have also seen an EOF error:

❯ DOCKER_VARIANTS=slim EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/arm64 mage package
--- Package Elastic-Agent
Downloading from https://artifacts-snapshot.elastic.co/cloudbeat/9.3.0-20f62a8a/downloads/cloudbeat/cloudbeat-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/prodfiler/9.3.0-457b6adf/downloads/prodfiler/pf-elastic-symbolizer-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/prodfiler/9.3.0-457b6adf/downloads/prodfiler/pf-elastic-collector-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/prodfiler/9.3.0-457b6adf/downloads/prodfiler/pf-host-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/apm-server/9.3.0-6d1a1b7b/downloads/apm-server/apm-server-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/fleet-server/9.3.0-cf7aa1c7/downloads/fleet-server/fleet-server-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/endpoint-dev/9.3.0-ca1e6b41/downloads/endpoint-dev/endpoint-security-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/beats/9.3.0-c080c72e/downloads/beats/agentbeat/agentbeat-9.3.0-SNAPSHOT-linux-arm64.tar.gz
Downloading from https://artifacts-snapshot.elastic.co/connectors/9.3.0-f726f49f/downloads/connectors/connectors-9.3.0-SNAPSHOT.zip
Done downloading connectors-9.3.0-SNAPSHOT.zip into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading pf-elastic-symbolizer-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading fleet-server-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading pf-elastic-collector-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading apm-server-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading pf-host-agent-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading endpoint-security-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading cloudbeat-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
Done downloading agentbeat-9.3.0-SNAPSHOT-linux-arm64.tar.gz into /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/archives/linux-arm64.tar.gz
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/pf-host-agent.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/apm-server.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/cloudbeat.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/endpoint-security.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/fleet-server.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/pf-elastic-collector.spec.yml does not exist, reverting to local specfile
>> File /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-drop/linux-arm64.tar.gz/pf-elastic-symbolizer.spec.yml does not exist, reverting to local specfile
>> BuildFleetCfg _meta/elastic-agent.fleet.yml to internal/pkg/agent/application/configuration_embed.go
>> BuildPGP from GPG-KEY-elasticsearch to internal/pkg/release/pgp.go
>> Building elastic-agent.yml for linux/amd64
>> Building elastic-agent.reference.yml for linux/amd64
>> Building elastic-agent.docker.yml for linux/amd64
--- CrossBuild Elastic-Agent
>> golangCrossBuild: Building for linux/arm64
>> Building using: cmd='build/mage-linux-arm64 golangCrossBuild', env=[GOARCH=arm64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-arm64]
--- Package artifact
>> package: Building linux-arm64
>> package: Building elastic-agent type=docker for platform=linux/arm64 fips=false
[+] Building 33.5s (22/22) FINISHED                                                          docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                         0.0s
 => => transferring dockerfile: 6.92kB                                                                       0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64"  0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64"  0.0s
 => [internal] load metadata for docker.io/redhat/ubi9-minimal:latest                                        0.3s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                        0.5s
 => [internal] load .dockerignore                                                                            0.0s
 => => transferring context: 2B                                                                              0.0s
 => [stage-1  1/11] FROM docker.io/redhat/ubi9-minimal:latest@sha256:6fc28bcb6776e387d7a35a2056d9d2b985dc4e  0.0s
 => => resolve docker.io/redhat/ubi9-minimal:latest@sha256:6fc28bcb6776e387d7a35a2056d9d2b985dc4e26031e98a2  0.0s
 => [internal] load build context                                                                            9.8s
 => => transferring context: 1.60GB                                                                          9.7s
 => [home 1/5] FROM cgr.dev/chainguard/wolfi-base:latest@sha256:42012fa027adc864efbb7cf68d9fc575ea45fe1b9fb  0.0s
 => => resolve cgr.dev/chainguard/wolfi-base:latest@sha256:42012fa027adc864efbb7cf68d9fc575ea45fe1b9fb0d166  0.0s
 => CACHED [home 2/5] RUN for iter in {1..10}; do         apk fix &&         apk add --no-cache shadow libc  0.0s
 => [home 3/5] COPY beat /usr/share/elastic-agent                                                            1.3s
 => [home 4/5] RUN true &&     chmod 0777 /usr/share/elastic-agent &&     mkdir -p /usr/share/elastic-agent  1.8s
 => [home 5/5] RUN setcap =p /usr/share/elastic-agent/data/elastic-agent-80a396/elastic-agent && true        0.4s
 => CACHED [stage-1  2/11] RUN for iter in {1..10}; do           microdnf update -y &&           microdnf i  0.0s
 => CACHED [stage-1  3/11] RUN set -e ;   TINI_BIN="";   TINI_SHA256="";   TINI_VERSION="v0.19.0";   case "  0.0s
 => CACHED [stage-1  4/11] COPY docker-entrypoint /usr/local/bin/docker-entrypoint                           0.0s
 => CACHED [stage-1  5/11] RUN groupadd --gid 1000 elastic-agent &&     useradd -M --uid 1000 --gid 1000 --  0.0s
 => [stage-1  6/11] COPY --chown=elastic-agent:elastic-agent --from=home /usr/share/elastic-agent /usr/shar  0.7s
 => [stage-1  7/11] RUN chmod 0777 /usr/share/elastic-agent &&     usermod -d /usr/share/elastic-agent elas  0.3s
 => [stage-1  8/11] RUN mkdir /licenses                                                                      0.1s
 => [stage-1  9/11] COPY --from=home /usr/share/elastic-agent/LICENSE.txt /licenses                          0.0s
 => [stage-1 10/11] COPY --from=home /usr/share/elastic-agent/NOTICE.txt /licenses                           0.0s
 => [stage-1 11/11] WORKDIR /usr/share/elastic-agent                                                         0.0s
 => exporting to image                                                                                      17.7s
 => => exporting layers                                                                                     15.5s
 => => exporting manifest sha256:d50411ff7a9000fa65e7a1634af2db2d6b3455ecfbde591d868c155f6162bbdb            0.0s
 => => exporting config sha256:6bd4fbc9ea3804d3ea075386cf41663a20792c43cc8a373f3f59c5391717467d              0.0s
 => => exporting attestation manifest sha256:2ef9a3904c6f74a77f41cb82588d8d8892c6d5058ca1d178c195b50d54851c  0.0s
 => => exporting manifest list sha256:e2c9b493a0a5003014177a3de8b95a1dd5fe582d2590623358d8bd3d1e40fef5       0.0s
 => => naming to docker.elastic.co/elastic-agent/elastic-agent-slim:9.3.0-SNAPSHOT                           0.0s
 => => unpacking to docker.elastic.co/elastic-agent/elastic-agent-slim:9.3.0-SNAPSHOT                        2.2s

 2 warnings found (use docker --debug to expand):
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64" (line 44)
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64" (line 5)
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
package ran for 1m19.166894541s
Error: error running package_test.go: running "go test /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go -root-owner -files /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/* --source-root /Users/cmackenzie/go/src/github.com/elastic/elastic-agent" failed with exit code 1, stdout: --- FAIL: TestDocker (0.94s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (0.94s)
        package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
        package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: unexpected EOF
FAIL
FAIL    com

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

❯ DOCKER_VARIANTS=slim EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/arm64 mage -v package
Running target: Package
--- Package Elastic-Agent
--- Packaging dependenciesVersion[9.3.0-SNAPSHOT], [{Name:linux/arm64 
...

>> build: Building elastic-agent-linux-arm64
exec: git "rev-parse" "HEAD"
Adding build environment vars: map[CGO_ENABLED:1]
exec: go "build" "-o" "/go/src/github.com/elastic/elastic-agent/build/golang-crossbuild/elastic-agent-linux-arm64" "-buildmode" "pie" "-trimpath" "-ldflags" "-s -X github.com/elastic/elastic-agent/version.buildTime=2025-12-04T15:16:15Z -X github.com/elastic/elastic-agent/version.commit=80a39611a93d307a5ca0c201e08eb6d826a7a852 -X github.com/elastic/elastic-agent/internal/pkg/release.snapshot=true" "github.com/elastic/elastic-agent"

...

 => [internal] load build definition from Dockerfile                                                         0.0s
 => => transferring dockerfile: 6.92kB                                                                       0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64"  0.0s
 => WARN: FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64"  0.0s
 => [internal] load metadata for docker.io/redhat/ubi9-minimal:latest                                        0.7s
 => [internal] load metadata for cgr.dev/chainguard/wolfi-base:latest                                        0.6s
 => [auth] redhat/ubi9-minimal:pull token for registry-1.docker.io                                           0.0s
 => [internal] load .dockerignore                                                                            0.0s
 => => transferring context: 2B                                                                              0.0s
 => [stage-1  1/11] FROM docker.io/redhat/ubi9-minimal:latest@sha256:6fc28bcb6776e387d7a35a2056d9d2b985dc4e  0.0s
 => => resolve docker.io/redhat/ubi9-minimal:latest@sha256:6fc28bcb6776e387d7a35a2056d9d2b985dc4e26031e98a2  0.0s
 => [internal] load build context                                                                            9.9s
 => => transferring context: 1.60GB                                                                          9.8s
 => [home 1/5] FROM cgr.dev/chainguard/wolfi-base:latest@sha256:42012fa027adc864efbb7cf68d9fc575ea45fe1b9fb  0.0s
 => => resolve cgr.dev/chainguard/wolfi-base:latest@sha256:42012fa027adc864efbb7cf68d9fc575ea45fe1b9fb0d166  0.0s
 => CACHED [home 2/5] RUN for iter in {1..10}; do         apk fix &&         apk add --no-cache shadow libc  0.0s
 => [home 3/5] COPY beat /usr/share/elastic-agent                                                            1.2s
 => [home 4/5] RUN true &&     chmod 0777 /usr/share/elastic-agent &&     mkdir -p /usr/share/elastic-agent  1.6s
 => [home 5/5] RUN setcap =p /usr/share/elastic-agent/data/elastic-agent-80a396/elastic-agent && true        0.4s
 => CACHED [stage-1  2/11] RUN for iter in {1..10}; do           microdnf update -y &&           microdnf i  0.0s
 => CACHED [stage-1  3/11] RUN set -e ;   TINI_BIN="";   TINI_SHA256="";   TINI_VERSION="v0.19.0";   case "  0.0s
 => CACHED [stage-1  4/11] COPY docker-entrypoint /usr/local/bin/docker-entrypoint                           0.0s
 => CACHED [stage-1  5/11] RUN groupadd --gid 1000 elastic-agent &&     useradd -M --uid 1000 --gid 1000 --  0.0s
 => [stage-1  6/11] COPY --chown=elastic-agent:elastic-agent --from=home /usr/share/elastic-agent /usr/shar  0.9s
 => [stage-1  7/11] RUN chmod 0777 /usr/share/elastic-agent &&     usermod -d /usr/share/elastic-agent elas  0.3s
 => [stage-1  8/11] RUN mkdir /licenses                                                                      0.1s
 => [stage-1  9/11] COPY --from=home /usr/share/elastic-agent/LICENSE.txt /licenses                          0.0s
 => [stage-1 10/11] COPY --from=home /usr/share/elastic-agent/NOTICE.txt /licenses                           0.0s
 => [stage-1 11/11] WORKDIR /usr/share/elastic-agent                                                         0.0s
 => exporting to image                                                                                      17.5s
 => => exporting layers                                                                                     15.0s
 => => exporting manifest sha256:61cb7da52a34fef41df964b66ac6f77b5a9c9ee5b7d7bb432c9f042d706d0500            0.0s
 => => exporting config sha256:9f2133d97a552a01c08ff42aaca80ba104ceee4866e0de5e22c7135178831515              0.0s
 => => exporting attestation manifest sha256:47920ceef1c3e0ebe3ba5cf581e6f2c9ae565c8c9a9151e4d943fd44df6fb3  0.0s
 => => exporting manifest list sha256:a7ee093275e4c412f21d658cd546ccaadd1d34cbbf9ff62f5e1305d1478e53f6       0.0s
 => => naming to docker.elastic.co/elastic-agent/elastic-agent-slim:9.3.0-SNAPSHOT                           0.0s
 => => unpacking to docker.elastic.co/elastic-agent/elastic-agent-slim:9.3.0-SNAPSHOT                        2.4s

 2 warnings found (use docker --debug to expand):
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64" (line 5)
 - FromPlatformFlagConstDisallowed: FROM --platform flag should not use constant value "linux/arm64" (line 44)
>>>> saving docker image docker.elastic.co/elastic-agent/elastic-agent-slim:9.3.0-SNAPSHOT to build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
Running dependency: TestPackages
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
exec: go "test" "-v" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go" "-root-owner" "-files" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/*" "--source-root" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent"
=== RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: archive/tar: invalid tar header
--- FAIL: TestDocker (0.98s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (0.98s)
FAIL
FAIL    command-line-arguments  1.308s
FAIL
package ran for 1m20.33738925s
Error: error running package_test.go: running "go test -v /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go -root-owner -files /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/* --source-root /Users/cmackenzie/go/src/github.com/elastic/elastic-agent" failed with exit code 1, stdout: === RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: archive/tar: invalid tar header
--- FAIL: TestDocker (0.98s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (0.98s)
FAIL
FAIL    command-line-arguments  1.308s
FAIL

❯ fd tar build
build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz.sha512
build/package/elastic-agent-darwin-arm64.tar.gz/
build/package/elastic-agent-linux-arm64.tar.gz/

❯ cd build/distributions
❯ tar xvfz ./elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
x blobs/
x blobs/sha256/
x blobs/sha256/15f4b55884339bddb52ca0550d5b1208576ecf31649e067b468dc4e7b90745bd
x blobs/sha256/2660c154ec15402bf6ed9903ff6284656dd00cc1ef2b63f9f0bad70ebfbeb19a
x blobs/sha256/47920ceef1c3e0ebe3ba5cf581e6f2c9ae565c8c9a9151e4d943fd44df6fb390
x blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
x blobs/sha256/5fc829be725016964a53962c0899b3904f61e7ea2cf2f993975a903c353e008c
x blobs/sha256/6109d8cc853ca48b2ef6ef609edcd5eeb577b4624e5d842efb36397ea5ab0670
x blobs/sha256/61cb7da52a34fef41df964b66ac6f77b5a9c9ee5b7d7bb432c9f042d706d0500
x blobs/sha256/71c28538f6153a9b14b9514606c5dc9dbe71e2ec7094530926017b5c3e2619f6
x blobs/sha256/82f3825ab4e5da669ecaae414db0fc7461459081a8fec1424201f786c02ac87a
x blobs/sha256/9ac290964a39cac1fae20154d4243aaaa89ca58b202b6687778a7afd6d57af33
x blobs/sha256/9f2133d97a552a01c08ff42aaca80ba104ceee4866e0de5e22c7135178831515
x blobs/sha256/a6633daac4718be0a225623fc2be585207dc92d9c75677ab0f3b4bff6ca0eddd
x blobs/sha256/a7ee093275e4c412f21d658cd546ccaadd1d34cbbf9ff62f5e1305d1478e53f6
x blobs/sha256/a8c045da7a498501e35530ccd5b230eef64ab28e750798502419116c7d22f3ce
x blobs/sha256/afdacd77d414ddf04e7b5b9af964f7e245bd4d28298a96843ee86635053d4103
x blobs/sha256/bf6bdd9cafaa66a1daaf22f1be003e7bd21bd20cb7e864e0276aff25a6368349
x blobs/sha256/f381df8dc69e51e03dc5c4ecc8f72f40e81fd4eb467163bcfd32ac5cf0d19fd0
x index.json
x manifest.json
x oci-layout

The tar command thinks the .tar.gz is valid but our tests don't.

[cmacknz]

With the output from the build above:

❯ fd tar build
build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz.sha512
build/package/elastic-agent-darwin-arm64.tar.gz/
build/package/elastic-agent-linux-arm64.tar.gz/

The TestPackages target will consistently fail but the tar command won't:

❯ DOCKER_VARIANTS=slim EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/arm64 mage -v TestPackages
Running target: TestPackages
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
exec: go "list" "-m"
Found Elastic Beats dir at /Users/cmackenzie/go/src/github.com/elastic/elastic-agent
exec: go "test" "-v" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go" "-root-owner" "-files" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/*" "--source-root" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent"
=== RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: archive/tar: invalid tar header
--- FAIL: TestDocker (1.00s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (1.00s)
FAIL
FAIL    command-line-arguments  1.350s
FAIL
Error: error running package_test.go: running "go test -v /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go -root-owner -files /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/* --source-root /Users/cmackenzie/go/src/github.com/elastic/elastic-agent" failed with exit code 1, stdout: === RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:426: error reading file /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz: archive/tar: invalid tar header
--- FAIL: TestDocker (1.00s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (1.00s)
FAIL
FAIL    command-line-arguments  1.350s
FAIL

[cmacknz]

With the changes in #11605 I get a more specific error:

❯ DOCKER_VARIANTS=slim EXTERNAL=true SNAPSHOT=true PACKAGES=docker PLATFORMS=linux/arm64 mage -v TestPackages
Running target: TestPackages
--- TestPackages, the generated packages (i.e. file modes, owners, groups).
--- TestPackages
>> Testing package contents
exec: go "list" "-m"
Found Elastic Beats dir at /Users/cmackenzie/go/src/github.com/elastic/elastic-agent
exec: go "test" "-v" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go" "-root-owner" "-files" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/*" "--source-root" "/Users/cmackenzie/go/src/github.com/elastic/elastic-agent"
=== RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:424:
                Error Trace:    /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go:1121
                                                        /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go:424
                                                        /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/
dev-tools/packaging/testing/package_test.go:127
                Error:          Received unexpected error:
                                archive/tar: invalid tar header
                Test:           TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
--- FAIL: TestDocker (1.02s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (1.02s)
FAIL
FAIL    command-line-arguments  1.427s
FAIL
Error: error running package_test.go: running "go test -v /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go -root-owner -files /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/* --source-root /Users/cmackenzie/go/src/github.com/elastic/elastic-agent" failed with exit code 1, stdout: === RUN   TestRPM
--- PASS: TestRPM (0.00s)
=== RUN   TestDeb
--- PASS: TestDeb (0.00s)
=== RUN   TestTar
--- PASS: TestTar (0.00s)
=== RUN   TestZip
--- PASS: TestZip (0.00s)
=== RUN   TestDocker
=== RUN   TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:126: /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/build/distributions/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
    package_test.go:424:
                Error Trace:    /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go:1121
                                                        /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go:424
                                                        /Users/cmackenzie/go/src/github.com/elastic/elastic-agent/dev-tools/packaging/testing/package_test.go:127
                Error:          Received unexpected error:
                                archive/tar: invalid tar header
                Test:           TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz
--- FAIL: TestDocker (1.02s)
    --- FAIL: TestDocker/elastic-agent-slim-9.3.0-SNAPSHOT-linux-arm64.docker.tar.gz (1.02s)
FAIL
FAIL    command-line-arguments  1.427s
FAIL

elastic-agent/dev-tools/packaging/testing/package_test.go

Lines 1119 to 1121 in 17cf103

	case slices.Contains(manifest.Layers, header.Name):
	layer, err := readTarContents(header.Name, tarReader)
	require.NoError(t, err)

[cmacknz]

Growing suspicion this is docker version dependent, I reproduce with:

❯ docker version
Client:
 Version:           28.5.2
 API version:       1.51
 Go version:        go1.25.3
 Git commit:        ecc6942
 Built:             Wed Nov  5 14:42:30 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.51.0 (210443)
 Engine:
  Version:          28.5.2
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.25.3
  Git commit:       89c5e8f
  Built:            Wed Nov  5 14:44:06 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.7.29
  GitCommit:        442cb34bda9a6a0fed82a2ca7cade05c5c749582
 runc:
  Version:          1.3.3
  GitCommit:        v1.3.3-0-gd842d771
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[cmacknz]

Two reports it doesn't reproduce with:

➜  elastic-agent git:(main) ✗ docker version
Client: Docker Engine - Community
 Version:           29.1.1
 API version:       1.52
 Go version:        go1.25.4
 Git commit:        0aedba5
 Built:             Fri Nov 28 11:33:04 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.1.1
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.4
  Git commit:       9a84135
  Built:            Fri Nov 28 11:33:04 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[pchila]

Failing images seem to contain gzipped layers, for example

➜  elastic-agent-8.19.9-SNAPSHOT-linux-amd64.docker git:(main) ✗ find . | xargs -I {} file {}
.: directory
./oci-layout: JSON data
./manifest.json: JSON data
./blobs: directory
./blobs/sha256: directory
./blobs/sha256/1878522322e0cc73b8aadbed17c056e17901fce2538cd6fd36db65d03f0e7905: JSON data
./blobs/sha256/093100a0b9a699e9ddf0c4787b98a26c2aefeebd5452b5f7b651682003724a72: gzip compressed data, original size modulo 2^32 5433856
./blobs/sha256/d68b3a5ab1577a588daa80c5741de9c4e7d71cdce6f0cc932a55bc19062cf4ce: gzip compressed data, original size modulo 2^32 6144
./blobs/sha256/f9dac9df514213fc0daa81ee729390aed9b9abfa39e2d41965530f42af996409: gzip compressed data, original size modulo 2^32 14848
./blobs/sha256/e5dbb7d9c42aca7734f6eb50ac59a30c348af0bc721fffbc590d2683e489dd00: gzip compressed data, original size modulo 2^32 26624
./blobs/sha256/55cc3155f500c3173a7d006755184c181cb058bed68d9fb5ea2e29a8ec45c2e2: JSON data
./blobs/sha256/c434a597767fa05df8393df5d6794e1744fc008418fcbdffa9f5eac5316ee850: gzip compressed data, original size modulo 2^32 1845391872
./blobs/sha256/7703d47f233347c05b327053d71fb2969345aa7c933acfcddf93ab829f74c0ab: gzip compressed data, original size modulo 2^32 1536
./blobs/sha256/9d6fa30021010e5ee61dac059ba8543b3b1f9037bc62507971c35434271b1ba6: gzip compressed data, original size modulo 2^32 6144
./blobs/sha256/2545bc41c7375f5989f48437c759a616ca04c3416b22ae8405b6e096801cf187: gzip compressed data, original size modulo 2^32 3584
./blobs/sha256/20043066d3d5c78b45520c5707319835ac7d1f3d7f0dded0138ea0897d6a3188: gzip compressed data, was "e8bce0aabd687e9ee90e0bada33884f40b277196f72aac9934357472863a80ae.tar", last modified: Thu Oct 16 19:23:04 2025, max compression, original size modulo 2^32 80635392
./blobs/sha256/eb73fd638c0ce0844ddb50a24bdd7b7ab47f2011f2e37eef908399cd7b0ac941: JSON data
./blobs/sha256/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: gzip compressed data, truncated
./blobs/sha256/24cb0fabf424a9f8dae31a4764a8bb1f73484a37a9fef05ebbb68f113ef9790e: JSON data
./blobs/sha256/9cc858581ed02db856f29d94c933999b7c2d62837b5b8e1b57b1d20e2db4fbc5: gzip compressed data, original size modulo 2^32 86253056
./blobs/sha256/05eb41c5c43e88b1f416a535a3645cd706d0daeecc71c9f0a02bb0abd877577d: gzip compressed data, original size modulo 2^32 17408
./blobs/sha256/f658398687ba0a78815962367f32eac4cc30daa87cbfa1d716ffcfd8827863a9: JSON data
./blobs/sha256/37f80932249b6afca23fb0bb6d97d08e8cbbe3fa1a7fb87a037009a464f22d8c: JSON data
./elastic-agent-8.19.9-SNAPSHOT-linux-amd64.docker.tar.gz: gzip compressed data, original size modulo 2^32 593796608
./index.json: JSON data

while images that do not fail package tests have layers that are plain tar data

➜  elastic-agent-slim-9.3.0-SNAPSHOT-linux-amd64.docker git:(main) ✗ find . | xargs -I {} file {}
.: directory
./oci-layout: JSON data
./manifest.json: JSON data
./repositories: JSON data
./blobs: directory
./blobs/sha256: directory
./blobs/sha256/a82670ceb3bbcbaaff5b1e6db6810237617bd4e8b32f83b41dea1c4f177cf5b6: JSON data
./blobs/sha256/cda6626c5d7821620039bcf60d84e8de84df4ce76688f97edd70c2acdf4d484a: POSIX tar archive
./blobs/sha256/8dd6bf74c654faba35b23a2bd092b521b702cc224995ca7fb6a0e98ddda6a5f0: JSON data
./blobs/sha256/762516c82416feea088a45501988003e80e907560d80a10d9b3d1982a3cfe9bc: POSIX tar archive
./blobs/sha256/518dc13936c82aa2d49b7e4197c7192058fc455ab4247d0f5cc45a659398162f: POSIX tar archive
./blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef: data
./blobs/sha256/0c17421bc6f9e84a73a9954f989297fe679746dbba448938d75b921628f933d0: JSON data
./blobs/sha256/cee8bfab88d77f3e694003b2649886cf667fd56cdb5290e8a99a41116b7310d6: JSON data
./blobs/sha256/04b9ce08aebfd81369998f748a58386fa968e883f19efe09b292baf934997a7f: JSON data
./blobs/sha256/55d5530fe8468c8c9907e0aa1de030811941604fa5f46de3db6dc15ec40906dd: POSIX tar archive
./blobs/sha256/c0ba4c793b329c1322641a933d44c0a620aae868721ba71021660add3389f87f: POSIX tar archive
./blobs/sha256/a57b05a391f427d374d3a8f56b84cdcae32efcbb0c0a8e1e1e77209b0beed5ca: POSIX tar archive
./blobs/sha256/2439eb38d163c4a10eeffe6ce830bbe5207d4814bfbd47d623cbe1b70d393ddd: JSON data
./blobs/sha256/5c2ba418b95e0b216980beeea8b1f72fb67690147303ea576e5e119e12a870c2: JSON data
./blobs/sha256/65d19504dfe0adc72a3e0f8d85fc0103132a959d16f7177344a96c30f866e66f: POSIX tar archive
./blobs/sha256/101e8afa748123edb4f6adc2779809b8d828a980f70f995b221b9c3b52e0e3a4: JSON data
./blobs/sha256/baa75364626aa6fface6015b03d24dbf03fd7b61f45b19aab8db4dbd37bc4ee7: JSON data
./blobs/sha256/a9ce1a1a68247cf261d603b56f6d49170a3236e6790ac5b209b0cb39239b2425: POSIX tar archive
./blobs/sha256/27c3052225b72d03f3fe7db40abfb61fb8b5bcf1ba890737c5d735b716830a94: POSIX tar archive
./blobs/sha256/2b6a421a3bc8a5fa5355794b8d7d21cb4f6218a87fd455aff98d9d4e4f5d4053: JSON data
./blobs/sha256/e239a4e5bef95ce322e147d7bb5c2b7ca814a95cbca708a05d0fcd57bb490cac: JSON data
./blobs/sha256/21c6d03e81223d4622ec26fb44a76c818665517432df274d9e1d735ee8fcd09e: JSON data
./blobs/sha256/b74f70ca7ad202332853f87601e82b6b40105cc156ffb37409ab1c210bc81955: POSIX tar archive
./blobs/sha256/e0ea8950f6739c5f398f9b4edad8b3447d62414f7c9e6a16eab55c14a19a7dc8: JSON data
./elastic-agent-slim-9.3.0-SNAPSHOT-linux-amd64.docker.tar.gz: gzip compressed data, original size modulo 2^32 1244114944
./index.json: JSON data
➜

This is probably a change in format depending on the specific version of docker, trying to figure out how to temporarily disable it.

[swiatekm]

Two reports it doesn't reproduce with:

➜  elastic-agent git:(main) ✗ docker version
Client: Docker Engine - Community
 Version:           29.1.1
 API version:       1.52
 Go version:        go1.25.4
 Git commit:        0aedba5
 Built:             Fri Nov 28 11:33:04 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.1.1
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.4
  Git commit:       9a84135
  Built:            Fri Nov 28 11:33:04 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Can confirm it doesn't reproduce for me on

Client:
 Version:           29.1.1
 API version:       1.52
 Go version:        go1.25.4 X:nodwarf5
 Git commit:        0aedba58c2
 Built:             Fri Nov 28 14:28:26 2025
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          29.1.1
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.4 X:nodwarf5
  Git commit:       9a84135d52
  Built:            Fri Nov 28 14:28:26 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41.m
 runc:
  Version:          1.4.0
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[v1v]

That was my thinking too, that a new version was released two weeks ago:

• https://docs.docker.com/desktop/release-notes/#4530

What pipelines did run that catch those failures? Can we enable that in the CI to run if any changes in the .buildkite pipelines?

---

No worries, I've found a way to test the packaging on PRs too:

• ci(packaging): let's test the packaging pipeline on PRs too #11612