Authentication to Azure Container Registry not working

[uveta]

Describe the Issue
We are using .dockerconfigjson secrets to store credentials to private Azure Container Registry. After upgrade from 1.4.16 to 1.6.0, pods are unable to authenticate to ACR. Error from pod's logs is:

no username present in auth config for server: {REDACTED}.azurecr.io

Steps To Reproduce

1. Use virtual kubelet 1.6.0
2. Store ACR credentials in a dockerconfigjson secret, e.g. regcred
3. Configure pods to use ACR credentials via

imagePullSecrets:
- name: regcred

4. Trigger scaling event to create pods
5. Pods are created but report authentication issue

Expected behavior
Pod authenticates to private ACR and ACI is created.

Virtual-kubelet version
1.6.0

Kubernetes version
1.27.3

Additional context
Works as expected when reverted back to 1.4.16

[smritidahal653]

Hi, thank you for reaching out to us. I followed the steps to reproduce but I am seeing the pods running successfully. Can you please provide additional logs and your workload yaml files if possible?

[uveta]

I tried to upgrade to version 1.6.0. again. Here is the log from virtual-kubelet pod, after I triggered scaling event:

Flag --provider has been deprecated, only one provider is supported
Flag --namespace has been deprecated, cannot set namespace, all namespaces watched
time="2023-09-19T12:45:45Z" level=info msg="features [init-container confidential-compute events] enabled"
I0919 12:45:45.582471       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca::/etc/kubernetes/certs/ca.crt"
time="2023-09-19T12:45:45Z" level=info msg="Pod cache in-sync"
time="2023-09-19T12:45:45Z" level=info msg="no container groups found for resource group [REDACTED]" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=aci.GetPods
time="2023-09-19T12:45:45Z" level=info msg="starting workers"
time="2023-09-19T12:45:45Z" level=info msg="started workers"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=5ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:03.879124929 +0000 UTC m=+258.337271955" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=0 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=3
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=5ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:03.879124929 +0000 UTC m=+258.337271955" queue=syncPodsFromKubernetes requeues=0 workerId=3
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=5ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:03.878653595 +0000 UTC m=+258.336800621" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=0 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=49
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=5ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:03.878653595 +0000 UTC m=+258.336800621" queue=syncPodsFromKubernetes requeues=0 workerId=49
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=10ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:04.025339955 +0000 UTC m=+258.483487081" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=1 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=2
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=10ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:04.025339955 +0000 UTC m=+258.483487081" queue=syncPodsFromKubernetes requeues=1 workerId=2
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=10ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:04.02472411 +0000 UTC m=+258.482871236" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=1 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=4
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=10ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:04.02472411 +0000 UTC m=+258.482871236" queue=syncPodsFromKubernetes requeues=1 workerId=4
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=20ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:04.097703171 +0000 UTC m=+258.555850297" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=2 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=6
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=20ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:04.097703171 +0000 UTC m=+258.555850297" queue=syncPodsFromKubernetes requeues=2 workerId=6
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=20ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:04.09659629 +0000 UTC m=+258.554743416" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=2 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=5
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=20ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:04.09659629 +0000 UTC m=+258.554743416" queue=syncPodsFromKubernetes requeues=2 workerId=5
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=40ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:04.166406097 +0000 UTC m=+258.624553223" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=3 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=8
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=40ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:04.166406097 +0000 UTC m=+258.624553223" queue=syncPodsFromKubernetes requeues=3 workerId=8
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:04Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=40ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:04.165468229 +0000 UTC m=+258.623615355" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=3 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=7
time="2023-09-19T12:50:04Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=40ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:04.165468229 +0000 UTC m=+258.623615355" queue=syncPodsFromKubernetes requeues=3 workerId=7
time="2023-09-19T12:50:04Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:05Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=80ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:04.471881421 +0000 UTC m=+258.930028447" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=4 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=9
time="2023-09-19T12:50:05Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=80ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:04.471881421 +0000 UTC m=+258.930028447" queue=syncPodsFromKubernetes requeues=4 workerId=9
time="2023-09-19T12:50:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:05Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=80ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:04.679467552 +0000 UTC m=+259.137614678" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=4 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=10
time="2023-09-19T12:50:05Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=80ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:04.679467552 +0000 UTC m=+259.137614678" queue=syncPodsFromKubernetes requeues=4 workerId=10
time="2023-09-19T12:50:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:05Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=160ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:05.174709221 +0000 UTC m=+259.632856347" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=5 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=11
time="2023-09-19T12:50:05Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=160ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:05.174709221 +0000 UTC m=+259.632856347" queue=syncPodsFromKubernetes requeues=5 workerId=11
time="2023-09-19T12:50:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:05Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:05Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:05Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:05Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:05Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:05Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:05Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=160ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:05.35380969 +0000 UTC m=+259.811956716" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=5 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=12
time="2023-09-19T12:50:05Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=160ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:05.35380969 +0000 UTC m=+259.811956716" queue=syncPodsFromKubernetes requeues=5 workerId=12
time="2023-09-19T12:50:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:06Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=320ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:05.914746223 +0000 UTC m=+260.372893349" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=6 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=13
time="2023-09-19T12:50:06Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=320ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:05.914746223 +0000 UTC m=+260.372893349" queue=syncPodsFromKubernetes requeues=6 workerId=13
time="2023-09-19T12:50:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:06Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=320ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:06.311574858 +0000 UTC m=+260.769721984" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=6 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=14
time="2023-09-19T12:50:06Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=320ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:06.311574858 +0000 UTC m=+260.769721984" queue=syncPodsFromKubernetes requeues=6 workerId=14
time="2023-09-19T12:50:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:07Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=640ms key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:07.034790955 +0000 UTC m=+261.492937981" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=7 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=15
time="2023-09-19T12:50:07Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=640ms error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:07.034790955 +0000 UTC m=+261.492937981" queue=syncPodsFromKubernetes requeues=7 workerId=15
time="2023-09-19T12:50:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:07Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=640ms key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:07.433292612 +0000 UTC m=+261.891439738" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=7 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=16
time="2023-09-19T12:50:07Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=640ms error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:07.433292612 +0000 UTC m=+261.891439738" queue=syncPodsFromKubernetes requeues=7 workerId=16
time="2023-09-19T12:50:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:08Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=1.28s key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:08.67855584 +0000 UTC m=+263.136702966" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=8 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=17
time="2023-09-19T12:50:08Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=1.28s error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:08.67855584 +0000 UTC m=+263.136702966" queue=syncPodsFromKubernetes requeues=8 workerId=17
time="2023-09-19T12:50:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:09Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=1.28s key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:09.072884194 +0000 UTC m=+263.531031220" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=8 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=18
time="2023-09-19T12:50:09Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=1.28s error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:09.072884194 +0000 UTC m=+263.531031220" queue=syncPodsFromKubernetes requeues=8 workerId=18
time="2023-09-19T12:50:09Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:10Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:10Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:10Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:10Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:10Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:10Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:11Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=2.56s key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:11.273893486 +0000 UTC m=+265.732040612" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=9 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=19
time="2023-09-19T12:50:11Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=2.56s error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:11.273893486 +0000 UTC m=+265.732040612" queue=syncPodsFromKubernetes requeues=9 workerId=19
time="2023-09-19T12:50:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:11Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=2.56s key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:11.675298853 +0000 UTC m=+266.133445879" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=9 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=20
time="2023-09-19T12:50:11Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=2.56s error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:11.675298853 +0000 UTC m=+266.133445879" queue=syncPodsFromKubernetes requeues=9 workerId=20
time="2023-09-19T12:50:11Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:15Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:15Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:15Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:15Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:15Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:15Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:16Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=5.12s key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:16.447847852 +0000 UTC m=+270.905994878" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=10 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=21
time="2023-09-19T12:50:16Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=5.12s error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:16.447847852 +0000 UTC m=+270.905994878" queue=syncPodsFromKubernetes requeues=10 workerId=21
time="2023-09-19T12:50:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:16Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=5.12s key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:16.829691998 +0000 UTC m=+271.287839124" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=10 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=22
time="2023-09-19T12:50:16Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=5.12s error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:16.829691998 +0000 UTC m=+271.287839124" queue=syncPodsFromKubernetes requeues=10 workerId=22
time="2023-09-19T12:50:16Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:20Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:20Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:20Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:20Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:20Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:20Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:25Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:25Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:25Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:25Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:25Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:25Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:26Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=10.24s key=[REDACTED]-ff5cf4597-7ndhh method=createOrUpdatePod name=[REDACTED]-ff5cf4597-7ndhh namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" phase=Pending plannedForWork="2023-09-19 12:50:26.717498791 +0000 UTC m=+281.175645817" pod=[REDACTED]-ff5cf4597-7ndhh podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=11 uid=eb3c2534-c99c-424d-9a3a-fc4510f3ced5 workerId=23
time="2023-09-19T12:50:26Z" level=warning msg="requeuing \"[REDACTED]-ff5cf4597-7ndhh\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=10.24s error="failed to sync pod \"[REDACTED]-ff5cf4597-7ndhh\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-ff5cf4597-7ndhh method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.873653595 +0000 UTC m=+258.331800621" plannedForWork="2023-09-19 12:50:26.717498791 +0000 UTC m=+281.175645817" queue=syncPodsFromKubernetes requeues=11 workerId=23
time="2023-09-19T12:50:26Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-ff5cf4597-7ndhh\", UID:\"eb3c2534-c99c-424d-9a3a-fc4510f3ced5\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:27Z" level=info msg="Updated k8s pod status" addedViaRedirty=false delayedViaRateLimit=10.24s key=[REDACTED]-64779b77f9-nsm6s method=createOrUpdatePod name=[REDACTED]-64779b77f9-nsm6s namespace=[REDACTED] originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" phase=Pending plannedForWork="2023-09-19 12:50:27.104066479 +0000 UTC m=+281.562213505" pod=[REDACTED]-64779b77f9-nsm6s podPhase=Pending queue=syncPodsFromKubernetes reason=ProviderFailed requeues=11 uid=36aff320-74b6-4d18-baea-4acb43655f9f workerId=24
time="2023-09-19T12:50:27Z" level=warning msg="requeuing \"[REDACTED]-64779b77f9-nsm6s\" due to failed sync" addedViaRedirty=false delayedViaRateLimit=10.24s error="failed to sync pod \"[REDACTED]-64779b77f9-nsm6s\" in the provider: no username present in auth config for server: [REDACTED].azurecr.io" key=[REDACTED]-64779b77f9-nsm6s method=handleQueueItemObject originallyAdded="2023-09-19 12:50:03.874124929 +0000 UTC m=+258.332271955" plannedForWork="2023-09-19 12:50:27.104066479 +0000 UTC m=+281.562213505" queue=syncPodsFromKubernetes requeues=11 workerId=24
time="2023-09-19T12:50:27Z" level=info msg="Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"[REDACTED]\", Name:\"[REDACTED]-64779b77f9-nsm6s\", UID:\"36aff320-74b6-4d18-baea-4acb43655f9f\", APIVersion:\"v1\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Warning' reason: 'ProviderCreateFailed' no username present in auth config for server: [REDACTED].azurecr.io"
time="2023-09-19T12:50:30Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:30Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:30Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:30Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:30Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:30Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:36Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:36Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:36Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:36Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:36Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:36Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:41Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:41Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:41Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:41Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:41Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:41Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:45Z" level=info msg="no container groups found for resource group [REDACTED]" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=aci.GetPods
time="2023-09-19T12:50:46Z" level=error msg="failed to query Container Group [REDACTED]-ff5cf4597-7ndhh, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:46Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-ff5cf4597-7ndhh in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:46Z" level=info msg="pod [REDACTED]-ff5cf4597-7ndhh will skip pod status update" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:46Z" level=error msg="failed to query Container Group [REDACTED]-64779b77f9-nsm6s, not found" azure.region=[REDACTED] azure.resourceGroup=[REDACTED] method=GetContainerGroup
time="2023-09-19T12:50:46Z" level=warning msg="cannot fetch aci events for pod [REDACTED]-64779b77f9-nsm6s in namespace [REDACTED]" error="cg is not found" method=PodsTracker.processPodUpdates
time="2023-09-19T12:50:46Z" level=info msg="pod [REDACTED]-64779b77f9-nsm6s will skip pod status update" method=PodsTracker.processPodUpdates

Here is a snippet of our k8s workload

apiVersion: apps/v1
kind: Deployment
metadata:
    name: service-context-aci-queue
    namespace: app
    labels:
        service: service
        context: context
        queue: queue
        app: app
spec:
    selector:
        matchLabels:
            service: service
            context: context
            queue: queue
            app: app
    template:
        metadata:
            name: service-context-aci-queue
            namespace: app
            labels:
                service: service
                context: context
                queue: queue
                app: app
        spec:
            nodeSelector:
                kubernetes.io/role: agent
                beta.kubernetes.io/os: windows
                type: virtual-kubelet
            tolerations:
                - key: virtual-kubelet.io/provider
                  operator: Exists
                - key: azure.com/aci
                  effect: NoSchedule
            containers:
                - name: service-context
                  image: private.azurecr.io/app/service/context:latest
                  imagePullPolicy: Always
                  resources:
                      requests:
                          memory: "4Gi"
                          cpu: "1.0"
                      limits:
                          memory: "4Gi"
                          cpu: "1.0"
            imagePullSecrets:
                - name: regcred