Hosting WITs via OCI on GHCR is flaky

[alexcrichton]

Today we did a security release of Wasmtime which involves doing a lot of CI all at once. We had lots of flaky failures due to wkg being unable to download WITs, such as:

+ wkg get --format wit --overwrite wasi:cli@0.2.6 -o crates/wasi/src/p2/wit/deps/cli.wit
Getting wasi:cli@0.2.6...
2026-02-24T17:57:34.381595Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.595113Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.946847Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:36.054084Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
Error: registry error: HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)

Caused by:
    HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)

Some example logs are:

• https://github.com/bytecodealliance/wasmtime/actions/runs/22363293742/job/64722423879
• https://github.com/bytecodealliance/wasmtime/actions/runs/22363788079/job/64724254926
• https://github.com/bytecodealliance/wasmtime/actions/runs/22363534893/job/64723336690
• https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64716132428
• https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64722334417
• https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64722932225

I don't know if this was an outage on GitHub's side, or what this is related to. I wanted to raise this here though since wkg's primary suorce is, I believe, ghcr artifacts for WASI. This is partly why I was curious about adding artifacts as in my experience github release artifacts have been pretty reliable (not perfect by any means...).

Regardless I wanted to raise this in case others had thoughts on this.

[ricochet]

This is a related to but different from the error that was reported in #873

I think this problem is multi-fold:

• Network retries should be implemented in wkg.
• There appears to be a potential bug in wkg authz handling. It looks like a classic tz bug given the start/end time header.
• It is reasonable to expect a hosted OCI registry to have a sufficient SLA for WASI. However GitHub's operational status has been severely degraded the past two months and if it continues, we could consider finding more reliable hosting for OCI artifacts.
• This was run in a tight loop and was throttled, aggravated by the invalid token. I don't see where GitHub documents the pull rate limit (it's not in what I get when calling https://api.github.com/rate_limit) or documented at github-packages doc but I suspect fixing the token would significantly increase the rate limit.

[alexcrichton]

If this ends up mostly being an issue in wkg I think it's reasonable to chalk it up to that and stiack with ghcr for now, I mostly just wasn't sure if this was a wkg thing or a GitHub thing.