acr: verify pulling images from private ACR/GCHRI registries

[mkulke]

In theory a pull-image-secret should be passed down to the podvm. It looks like it doesn't work for ACR (see discussion here), so we should verify it for private ACR and GHCR registries and add a test-

Possibly, this test should be in image-rs (guest-components) instead in the long term.

[tak-ka3]

@mkulke
I've tested private registry authentication with ACR and can confirm it's working correctly in the current codebase.

Test Results:

• Successfully pulled private images from Azure Container Registry (ACR)
• imagePullSecrets were correctly processed and passed to PodVM
• auth.json was properly generated and transferred to the guest VM
• Image pull completed successfully (verified by SHA256 digest in imageID)

Test Environment:

• ACR: caatestphase1.azurecr.io
• Image: nginx:alpine (private)
• Runtime: kata-remote (PeerPods on Azure)
• Authentication: kubernetes.io/dockerconfigjson secret

Logs showing successful authentication:

[util/k8sops] Processing secret default/acr-secret (type: kubernetes.io/dockerconfigjson)
[util/k8sops]   Registry caatestphase1.azurecr.io: auth=true, username=true, password=true
[util/k8sops] Successfully collected credentials for 1 registries
[adaptor/cloud] Writing auth.json (138 bytes) to PodVM

Pod status:

$ kubectl get pod acr-test -o jsonpath='{.status.containerStatuses[0].imageID}'
caatestphase1.azurecr.io/nginx@sha256:7cf0c9cc3c6b7ce30b46fa0fe53d95bee9d7803900edb965d3995ddf9ae12d03

The SHA256 digest confirms the image was successfully pulled from the private ACR.

Note: This test was conducted with commit 546ae72 ("allow docker image pull secrets with username/password only") already merged, which may have resolved the original issue.

Recommendations:

1. Add E2E tests for ACR/GHCR private registry authentication
2. Test GHCR authentication as well (not yet verified)
3. Document the expected imagePullSecrets format

Would it be helpful to add automated tests for this scenario?

[mkulke]

thanks a lot for your effort @tak-ka3! it's good to know that we support private registries, we probably need to have a test for a private registry.

we had a discussion around this issue here: https://cloud-native.slack.com/archives/C04A2EJ70BX/p1764709701475639

specifically this comment from @bpradipt is relevant here:

{
    "auths": {
        "registry.." : {
         }
    }
}

seems to work, while that one doesn't:

{
  "registry...": {
   }
}

if both forms are common we probably want to fix that in guest-components (image-rs), since afaik we only pass the auth.json to image-rs.

If it's not that trivial, we could either silently (or with a warning) fix the format (not a fan of implicit changes in generaly, but I guess if we used some golang oci library it would be less arbitary), alternatively we could bail out if we encounter an unsupported format. I could imagine this is annoying though: an imagePullSecret might be apply to the confidential pod (e.g. because it's defined namespace-wide or something) but not required for the containers in the confidential pod and we'd bail out for no reason.

[tak-ka3]

Thank you @mkulke for the detailed explanation and context around this issue! The following is the result of investigation and the possible next step.

Investigation Results

I've investigated the current implementation in detail, and both formats are already supported ✅

Current Implementation (pkg/adaptor/k8sops/node.go)

CAA performs the following processing:

1. Input: Accepts both .dockerconfigjson and .dockercfg formats
2. Conversion: Both formats are internally unified into the Auths struct
3. Output: Always generates containers-auth.json format (with "auths" field)

Specifically, the code at node.go:160-172:

• Parses .dockerconfigjson by unmarshaling into Auths struct (which has "auths" field)
• Parses .dockercfg by unmarshaling directly into Registries map
• Both are merged into the same Auths struct
• Final output via json.Marshal(auths) always produces the containers-auth.json format

This means regardless of which format users provide, image-rs receives the correct format it expects.

The Real Issues

The functionality works correctly, but we're lacking:

1. ❌ Test Coverage: No dedicated E2E tests for ACR/GHCR
2. ❌ Documentation: Supported formats are not explicitly documented
3. ❌ Observability: Insufficient logging makes troubleshooting difficult

Proposed Solution

Since the code already handles both formats correctly, I propose to focus on:

Phase 1: Enhanced Observability

• Add detailed logging for imagePullSecrets processing
• Make it visible which registries and formats are being processed

Phase 2: Test Coverage

• E2E tests for ACR private image pull
• E2E tests for GHCR private image pull
• Unit tests for both formats and edge cases

Phase 3: Documentation

• Document supported formats explicitly

I'm happy to work on implementing these improvements.