issue with pulling Helm charts from private Docker Hub OCI registry

[kurtlieu]

I'm attempting to follow the ArgoCD app of apps pattern.
I have access to an external, private Docker Hub where the customer stores their Helm charts.
I have created a Kubernetes Secret with the credentials the customer has provided, and see that it connects in the ArgoCD portal under Repositories.
I can also manually run helm registry login registry-1.docker.io , authenticate, and pull their helm charts

helm pull oci://registry-1.docker.io/CUSTOMER_ORG_NAME/CUSTOMER-HELM-CHART
apiVersion: v1
kind: Secret
metadata:
  name: customer-helm-registry-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: customer-helm-registry
  url: registry-1.docker.io
  enableOCI: "true"
  type: helm
  username: "REDACTED"
  password: "REDACTED"
  ForceHttpBasicAuth: "true"

I have my own private GitHub repo where my ArgoCD Application manifests and (parent) Helm charts are stored.
in my Application manifest, I point to the GitHub repo itself

spec:
  project: default
  sources:
  - repoURL: [email protected]:my_org_github/aip.git  # Your IAC repo URL
    path: apps/MY_PARENT_HELM_CHART_WRAPPER_OR_CUSTOMER_HELM_CHARTS
    targetRevision: HEAD
    helm:
      # releaseName: customer_app_name
      passCredentials: true
      valueFiles:
        - dev.values.yaml

In my Chart.yaml, I have dependencies set to point to the customer's Docker Hub

dependencies:
- name: "CUSTOMER-HELM-CHART"
  version: 1.0.0

  # This auths successfully and the syntax to pull is correct, but does NOT work. It should auth to oci://registry-1.docker.io, NOT oci://registry-1.docker.io/CUSTOMER_ORG_NAME
  # repository: "oci://registry-1.docker.io/CUSTOMER_ORG_NAME"

  # This auths correctly, but does NOT pull the chart correctly (because I haven't specified the org)
  repository: "oci://registry-1.docker.io"

As pointed out in the comments, I've observed the above behavior based on the logs from the argocd-repo-server container
Changing the name to CUSTOMER_ORG/customer-helm-chart introduces this error
You may need to run helm dependency build to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: CUSTOMER_ORG/customer-helm-chart

I understand that ArgoCD itself supports the ability to authenticate and directly pull from OCI registries.
and that Helm dependencies can also pull from private OCI repositories.
but chaining this together where I introduce a middle/parent Helm chart, causes me this discrepancy.

Does anyone have insight as to whether or not this design is possible?
Whether I have missed any sort of configuration that would enable this, or whether it's a bug or edge case with ArgoCD?
Or am I chasing down a completely lost cause.

I would need to have the project's Helm charts stored within my own GitHub or have hosted in another private Helm Chart repository such as ChartMuseum or ArtifactHub

I saw this issue which seems similar, but I'm unclear as to where I might be going wrong
#12436

[rhysjtevans]

Bitnami have also moved to this pattern of hosting

[denniskorbginski]

I don't think this issue is related to the app-of-apps-pattern specifically. I'm trying to use my Docker Hub credentials to avoid running into unauthenticated image pull limits. Creating a simple app like this does not make use of my credentials:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bitnami-nginx-demo
spec:
  destination:
    namespace: bitnami-nginx-demo
    server: https://kubernetes.default.svc
  source:
    repoURL: registry-1.docker.io/bitnamicharts
    targetRevision: 19.0.1
    chart: nginx
    helm:
      passCredentials: true
  sources: []
  project: apps

I've created this secret, which also shows up fine in Argos list of repositories

apiVersion: v1
kind: Secret
metadata:
  name: dockerhub-oci-credentials-test
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  enableOCI: "true"
  name: "dockerhub"
  type: "helm"
  url: "registry-1.docker.io"
  username: "username"
  password: "password"
  ForceHttpBasicAuth: "true"

In the debug logs, I also find this line which makes me thing that Argo does realize that a secret for Docker Hub exists:

level=debug msg="registry-1.docker.io has credentials"

[knightprog]

Same problem on my side.
I managed to make the oci dependency to work with a private Dockerhub with a (really ugly) workaround by wrapping the helm binary into a shell script:

repoServer:
...
    initContainers:
    - name: ugly_docker_hub_login_workaround
...
      volumeMounts:
        - name: helm-custom-binary
          mountPath: /helm-custom-binary
      command: [ "/bin/bash", "-c" ]
      args:
        - |
          cp /usr/local/bin/helm /helm-custom-binary/helm.bin;
          cat <<'EOF' > /helm-custom-binary/helm
          #!/usr/bin/env bash
          args=($@)
          if [ "$#" -gt 2 ] && [ $1 == "registry" ] && [ $2 == "login" ] && [ $3 == "registry-1.docker.io/mycoolorg" ]; then
            args[2]="registry-1.docker.io"
          fi
          helm.bin ${args[@]}
          EOF
          chmod +x /helm-custom-binary/helm;
  volumeMounts:
    - name: helm-custom-binary
      mountPath: /usr/local/bin/helm
      subPath: helm
    - name: helm-custom-binary
      mountPath: /usr/local/bin/helm.bin
      subPath: helm.bin
  volumes:
    - name: helm-custom-binary
      emptyDir: {}
...