qemu-aarch64 nondeterministic crashes

[lts-rad]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

wanted to report that i recently switched a build pipeline running on ubuntu to the upstream ubuntu qemu-user-static. the upstream ubuntu qemu is further along on version number than this action btw, so that this saves people time.

we were witnessing a problem where qemu-aarch64 installed by this action would crash nondeterministically on github's x86-64 runners as well as on self-hosted runners using 12th gen and 13th gen intel systems. we dont have time to triage and diagnose the issue, because it's non deterministic its not clear what the underlying cause is.

Expected behaviour

qemu-aarch64 doesnt crash at random

Actual behaviour

qemu-aarch64 crashed at random

Repository URL

No response

Workflow run URL

No response

YAML workflow

n/a

Workflow logs

No response

BuildKit logs

Additional info

No response

[smoke]

We have a Ruby app depends on quite a bunch of "native" extensions packages - that are packages having to be built through gcc

Targeting linux/amd64,linux/arm64

we have consistent Gem::Ext::BuildError: ERROR: Failed to build gem native extension. caused by gcc: internal compiler error: Segmentation fault signal terminated program cc1
but that will affect anything built by gcc or similar (nodejs packages, etc.).

Similar thing happened with a Go project needed some stuff built with gcc.

I was suspecting this issue, but it turned out another action int128/create-ecr-repository-action@v1.342.0 is rogue, I am still puzzled how it could brake stuff in such way int128/create-ecr-repository-action@v1.342.0...v1.348.0

UPDATE: Still investigating, no clarity ... Issue is happening sporadically, combined with caching it is really difficult to get a good grip on what is the cause and what is a fix.

[lts-rad]

the qemu that is shipped by this action seemed broken the last time i used it. i recommend trying to use your base images qemu instead.

[smoke]

Indeed there are some very strange and sporadic qemu@v7 issues in combination with some other unidentified changes / inconsistencies (buildx, gh-actions, runners or whatever).
I have managed to reproduce locally.

Qemu v7 used to work well for quite some time but not anymore, since ~1-2 weeks it started resulting in sporadic but stubborn segmentation faults in gcc and other executables.

Having build cache lead this to be very difficult to identify and fix, so I am still not sure how stable the fix is, but
what seems to fix it so far is using qemu@v8 through:

      uses: docker/setup-qemu-action@v3.3.0
      with:
        image: tonistiigi/binfmt:qemu-v8.1.5
        platforms: ${{ inputs.platforms }}

[crazy-max]

As this is qemu related can you open an issue on https://github.com/tonistiigi/binfmt? Thanks

[smoke]

there is related ticket already tonistiigi/binfmt#215

[mayeut]

This does not only affect aarch64 running on x86_64 but also at least s390x & ppc64le.
All repos I have seen using setup-qemu-action with ubuntu-latest/ubuntu-24.04 started failing with the update of the runner this week.

@crazy-max, I understand the issue is upstream so not asking to re-open the issue but could it be at least pinned in order for users of the action to easily find the issue and workaround while waiting for a fix upstream ?

For aarch64, another solution is to switch from emulated to native with the availability of ubuntu-24.04-arm runners: https://github.blog/changelog/2025-01-16-linux-arm64-hosted-runners-now-available-for-free-in-public-repositories-public-preview/