fix registry authentication headers encoding

[oneingan]

i only check it with digitalocean private registry but i require to change the base64 encoding to make it work.

Also, during debug, i ve found a bug in ONLY_KNOWN_REGISTRIES logic.

[s4ke]

Are you sure that this fixes things? I am pretty sure the code already works as is?

With Docker Harbor we needed to use base64url as far as I remember. See moby/moby#33434

[s4ke]

I released as 0.9.3 for the ONLY_KNOWN_REGISTRIES part of this patch

[oneingan]

Yes, I’ve checked and double-checked with registry.digitalocean.com and ghcr.io. And strictly speaking, the header is not a URL.

Also, see: https://stackoverflow.com/a/50755411

[s4ke]

The header is not a URL, the encoding should be a base64url encoded string.

Can you produce a minimum reproducer? Or show me logs of an error?

[oneingan]

Sure!

Also, I’d like to check with you if the current behavior is desirable. Or maybe the proxy doesn’t need to strip the original headers when no auth overrides are set?

[s4ke]

The stripping of auth is intentional. I am open to have this as a feature flag, but for us a main pain point was the fact that if we deploy things with user credentials, then those user credentials are then stored in docker swarm. This makes it unpredictable to know which credentials are currently in use in a cluster. This makes things harder if you need to rotate a credential or offboard employees.

[oneingan]

The good one fails because it correctly uses the credentials, but they are invalid:

$ curl -v --silent --header "x-registry-auth: $XRA" --unix-socket /run/docker.sock "http://v1.41/distribution/ghcr.io/jonashackt/hello-world:latest/json"
*   Trying /run/docker.sock:0...
* Connected to v1.41 (/run/docker.sock) port 0
* using HTTP/1.x
> GET /distribution/ghcr.io/jonashackt/hello-world:latest/json HTTP/1.1
> Host: v1.41
> User-Agent: curl/8.11.1
> Accept: */*
> x-registry-auth: eyJ1c2VybmFtZSI6ImZha2V1c2VyIiwicGFzc3dvcmQiOiJmYWtlcGFzcyIsInNlcnZlcmFkZHJlc3MiOiJnaGNyLmlvIiwiZW1haWwiOiJqdWFuam8ucHJlc2FAbmFkYW5peC5jb20ifQ==
> 
* Request completely sent off
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.47
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/27.3.1 (linux)
< Date: Wed, 19 Feb 2025 16:42:20 GMT
< Content-Length: 98
< 
{"message":"Head \"https://ghcr.io/v2/jonashackt/hello-world/manifests/latest\": denied: denied"}
* Connection #0 to host v1.41 left intact

The bad one ignores the credentials completely and pulls the image anonymously:

$ curl -v --silent --header "x-registry-auth: $XRA" --unix-socket /run/docker.sock "http://v1.41/distribution/ghcr.io/jonashackt/hello-world:latest/json"
*   Trying /run/docker.sock:0...
* Connected to v1.41 (/run/docker.sock) port 0
* using HTTP/1.x
> GET /distribution/ghcr.io/jonashackt/hello-world:latest/json HTTP/1.1
> Host: v1.41
> User-Agent: curl/8.11.1
> Accept: */*
> x-registry-auth: eyJ1c2VybmFtZSI6ImZha2V1c2VyIiwicGFzc3dvcmQiOiJmYWtlcGFzcyIsInNlcnZlcmFkZHJlc3MiOiJnaGNyLmlvIiwiZW1haWwiOiJqdWFuam8ucHJlc2FAbmFkYW5peC5jb20ifQ
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Api-Version: 1.47
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/27.3.1 (linux)
< Date: Wed, 19 Feb 2025 16:43:20 GMT
< Content-Length: 230
< 
{"Descriptor":{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:bdf82ff4828a18fcf26b47f42e69f30caf5f33320b276ec93d16bf80019064fd","size":739},"Platforms":[{"architecture":"amd64","os":"linux"}]}
* Connection #0 to host v1.41 left intact

[s4ke]

please try to do this with the docker cli.

The docker cli uses base64url - pretty sure. But to be sure, please verify your hunch.

[oneingan]

How? Do you mean using docker login to create a ~/.docker/config.json? I don’t know the internals, but it seems very different from the auth process you’re using here.

[s4ke]

what i mean is: Create a swarmgate instance with a docker/config.json that contains the credentials.

Then create a context in your docker cli, and point it to swarmgate. Then try to create a service that uses your docker image on a private registry.

[oneingan]

Excuse me if I’m wrong, but this is exactly what I did before, and that’s where the error started. Then, after debugging with curl and testing /run/secrets/registry_auth_overrides, I discovered that the overrides were ignored. After investigating further, I found that the error was in the headers.

This was the original error when I tried to deploy a service:

│ Error: Error response from daemon: Permission check failed, Error: Head "https://registry.digitalocean.com/v2/mycompany/metis-worker/manifests/1c7efa5": unauthorized: authentication required

[s4ke]

I will verify the behavior on our end hopefully next week.

[oneingan]

Nice! I still pretty sure the change will fix the bug but no hurries because I'm running my own fork right now.

[oneingan]

do you achieve to verify the behavior finally?

[oneingan]

ping

[oneingan]

ping

[oneingan]

can we get this merged. This change its working for me during 8 months.