[Bug] SOCI 2nd Image pull not working with layer unavailable error

[prafgup]

Description

Hey Devs,
(I have used AI for some analysis and writeup, but have confirmed that things are correct)

When using soci-snapshotter-grpc with a private registry that uses Basic auth (username and token) and short-lived credentials (via CRI keychain, we use k8s Secrets and remove them once pod has finished), layers that were successfully pulled/partially pulled become unavailable hours later. The checkAvailability health check fails because the snapshotter cannot re-authenticate with the original credentials expire.

Environment

• soci-snapshotter version: 0.12.1
• Registry auth: Basic (WWW-Authenticate: Basic realm="XXXX")
• Credential source: CRI keychain (cri_keychain.enable_keychain = true)
• Container runtime: containerd

Behavior

First pull (works): Kubelet issues PullImage() → CRI keychain receives fresh credentials → SOCI artifacts fetched and layers mounted via FUSE successfully.

Hours later (fails): A new pod pulls the same image. checkAvailability runs on the existing FUSE-mounted layers, attempts to verify blob connectivity, receives 401, and tries to re-authenticate. Re-authentication fails with "failed to find supported auth scheme: not implemented", causing layers to be marked unavailable.

When a pod is launched and kubelet tries to pull the same image hours after on the same node I see this error -

failed to create containerd container: error unpacking image: failed to prepare extraction snapshot 
"extract-86525.... sha256:c49c6f8114925036148a3596d382b44b770483093594ba21be9923ea3d51a71c": 
layer "177" unavailable: unavailable

My soci config -

[cri_keychain]
enable_keychain = true
image_service_path = "/run/containerd/containerd.sock"

Heres a rough timeline of the error -

  ┌─────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │    Time     │                                                              Event                                                               │
  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 15:04:51    │ Image pull starts, snapshots prepared. Credentials in CRI keychain.                                                              │
  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 15:05:19-22 │ Remote snapshots (FUSE) successfully prepared for layers with ztocs (snapshots 14, 16, 17, 20). Auth works here.                 │
  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 15:05:49    │ Background fetcher kicks in (~27 seconds after pull). Tries to fetch span 0 from layer sha256:18d8de.... Gets 401 Unauthorized.  │
  │             │ "failed to find supported auth scheme: not implemented". CRI keychain credentials are already gone.                              │
  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 15:05:51-58 │ More background fetch auth failures for layers sha256:17431c..., sha256:ea2a54..., sha256:ac273d...                              │
  ├─────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 18:06:43    │ New pod tries to use same image. checkAvailability() runs → tries to connect to blobs → 401 on all 4 remote layers → "layer      │
  │             │ unavailable" → container creation fails                                                                                          │
  └─────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Error logs

  {"level":"WARN","message":"check failed","params":{"error":"failed(layer:\"sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd\",
  ref:\"registry.example.com/image:tag\"): unable to create remote fetcher: failed to redirect: request to registry failed: failed to handle challenge: failed to find     
  supported auth scheme: not implemented (host \"registry.example.com\", ref:\"registry.example.com/image:tag\",
  digest:\"sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd\"): failed to refresh connection"}}                                                     
                                                                  
  {"level":"WARN","message":"layer is unavailable","params":{"error":"failed(layer:\"sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd\",            
  ref:\"registry.example.com/image:tag\"): unable to create remote fetcher: failed to redirect: request to registry failed: failed to handle challenge: failed to find
  supported auth scheme: not implemented (host \"registry.example.com\", ref:\"registry.example.com/image:tag\",                                                           
  digest:\"sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd\"): failed to refresh connection"}}

  {"level":"WARN","message":"failed to refresh the layer \"sha256:17431c26b529de6c07033da89d4c81ddd0b76bbec8f6edc10c11d670cb6ff1db\" from                                  
  \"registry.example.com/image:tag\"","params":{"error":"unable to create remote fetcher: failed to redirect: request to registry failed: failed to handle challenge:
  failed to find supported auth scheme: not implemented (host \"registry.example.com\", ref:\"registry.example.com/image:tag\",                                            
  digest:\"sha256:17431c26b529de6c07033da89d4c81ddd0b76bbec8f6edc10c11d670cb6ff1db\")"}}


  Background fetcher also fails with the same auth issue (I can disable it but I dont think its going to help much):                                                                                                                  
  {"level":"WARN","message":"error trying to resolve layer, removing it from the queue","params":{"error":"error trying to fetch span with spanId = 0 from layerDigest =
  sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd: failed to handle challenge: failed to find supported auth scheme: not implemented"}} 

I have checked the config but I don't see any config that would help with it. Would appreciate your expertise over this. The credentials we provide to the cri keychain are short lived and removed/expired once the pod has finished. New credentials are injected when launching a new pod.

From AI -

  The full sequence:                                                                                                                                                       
                                                                  
  1. Pod A pulls image X → PullImage() intercepted → CRI keychain stores creds-A → RegistryManager creates AuthClient with creds-A, caches it permanently in               
  registryHostMap                                                 
  2. Hours pass → creds-A expire                                                                                                                                           
  3. Pod B pulls same image X → PullImage() intercepted → CRI keychain updated with fresh creds-B → but containerd sees image already exists, skips layer work
  4. Container starts → Mounts() → checkAvailability() → blob.Check() → 401                                                                                                
  5. Refresh path → getSources() → hosts(refspec) → cache hit → returns stale AuthClient with expired creds-A → re-auth fails → "layer is unavailable"                     
                                                                                                                                                                           
  The CRI keychain HAS fresh creds-B, but RegistryManager never asks for them again — it returns the cached RegistryHost from the first pull.

Containerd tries Prepare() → SOCI's mounts() → checkAvailability() → blob.Check() → 401 → layer unavailable

More AI analysis but sounds about right -

Steps to reproduce the bug

No response

Describe the results you expected

The 2nd pod should be able to launch sucessfully, the layer should not be marked unavailable. The layer check should authenticate with the new creds if required.

Host information

1. OS: x86_64
2. Snapshotter Version: 0.12.1
3. Containerd Version: v2.2.2

Any additional context or information about the bug

No response

[sondavidb]

The layer check should authenticate with the new creds if required.

We don't support this properly unfortunately :/ We re-auth with existing credentials but we don't have ways to properly rotate in new credentials as needed. (I need to remember why but I don't remember the reason for this right now.) You can see a bit of a related discussion I had here, but the TL;DR is that long-lived tasks like this can have issues with authentication.

With that said, "no basic auth" isn't an error I'd expect to see. So this issue isn't exactly 1:1 with the above. But I wonder if that has to do with the lack of credential rotation that I mentioned before.

FWIW, we do skip layer auth checks if the layer is fully downloaded (ref). If you're expecting to run tasks for multiple hours on a pod I would expect the background fetcher to have finished fetching all necessary spans at that point. But I suppose depending on network constraints and background fetcher config and image size we can't always expect it to be the case.

Related: #522 (file in question got moved here)

[prafgup]

@sondavidb Thanks for the explanation. We are using k8s Secrets for auth through CRI keychain, and once the pod is finished the Secret is deleted and the Token is expire soon after.
We see span fetch failing with -

  {"level":"WARN","message":"error trying to resolve layer, removing it from the queue","params":{"error":"error trying to fetch span with spanId = 0 from layerDigest = sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd: failed to handle challenge: failed to find supported auth scheme: not implemented"}} 

Which is fine we can disable span fetching. (we do use the auth schema (WWW-Authenticate: Basic realm="XXXX") which is same for this container registry and does not changes)

Wondering if there a way or hack to somehow invalidate the cached credentials in case there is any 401 error so it retries. Just to unblock myself (I don't think persisting secrets or not expiring creds is preferable for us).

Maybe can you take a look here if this makes sense as a sanity check (I will refine it if it looks promising) - #1925

• Basically deleting the cached creds so its pulled from k8s again in case of errors.

Or is there any better hack/approach we can use here to unblock our usecase? (Maybe deleting/disabling this whole caching mechanism?)

With that said, "no basic auth" isn't an error I'd expect to see. So this issue isn't exactly 1:1 with the above. But I wonder if that has to do with the lack of credential rotation that I mentioned before.

I believe this is due to the creds being expired (or soci caching empty creds due to retries somethow). Would need to dive deep here tho from my end.

[prafgup]

but the TL;DR is that long-lived tasks like this can have issues with authentication.

@sondavidb just to clarify this as well, my task is very short lived ~5-10sec and the pod is evicted as soon as it completes. (The full image is ~1.5gb) and soci did lazy load the parts here. But afterwards span fetching failed with the provided error as well as the next pull (after 3 hrs) failed.

[prafgup]

Just confirmed -

request to registry failed: failed to handle challenge: failed to find supported auth scheme: not implemented

curl -sv -u "valid-username:wrongpassword" https://registry.example.com/image/blobs/sha256:18d8de3e8af0f2fb1b2fca8bea9f20cc5d4c783192bcb6cad631b6fee0686bbd 

Valid repo username + wrong password → 401 with NO WWW-Authenticate header.

The actual issue is creds not being refreshed (as its using the old cached ones) #1925 should help. I will refine that PR once I get an ack :)

[sondavidb]

my task is very short lived ~5-10sec

Ah yeah sorry I used the wrong terminology. The task itself is not the problem, but the pod lifecycle, as it seems you've figured out. The credentials are cached on the pod on the first pull which means subsequent tasks landing on the pod will reuse those credentials, which is where we're seeing this error.

I'll take a look at the PR and leave some comments there for your fix in particular

[prafgup]

Some more issues and investigations posted here - #1927 (comment)

[prafgup]

Just to keep the thread updated, after #1927 (comment) I did test out #1932 and let it bake for few days, which seems to be working fine. Although is a bit complex overall and in the worst case leads to a lot of 401 calls (potentially trying all the credentials seen till now, we can optimize it a bit to try from latest added creds to oldest increasing the chance for a valid cred hit early on but would add more complexity).

I am exploring other solutions on our end for the timebeing but willing to refine and contribute this creds chain rotation mechanism if its deemed to provide value.

[prafgup]

So after a lot of experimentation on our end (with k8s creds rotation, cache invalidation, etc...), some of which might actually help soci a bit -

• Invalidate cached registry hosts to re-auth on layer check failure #1927
• [Testing] Rotate k8s creds #1932
• Invalidate host creds in case of failure and retry #1925
• [Extension] Refresh hosts/sources on Mount failures instead of using cached creds prafgup/soci-snapshotter#2
• [EXTENSION] Fall back to current pull's image ref for auth when shared base layer credentials are revoked prafgup/soci-snapshotter#3

but we have settled upon increasing the validity of the token being used for soci image pulling + enabled background fetching.
We had some more ideas (wrt this issue as well as more features) on our end but that would be more custom to our infrastructure and making it generic would be difficult, thus in long term we might be moving towards maintaining a internal fork of soci.

Thus if required we feel free to close this ticket, or use this ticket for making the creds caching better in future :)