Extend HTTPS proxy support for SSH for fleet.yaml's helm.chart field

[p-se]

Background

Fleet's fleet.yaml supports a helm.chart field that can reference a Helm chart in a git repository. Downloading these charts is handled by go-getter, which currently shells out to the system git binary for git URLs.

SSH through an HTTPS proxy (introduced for GitRepo.spec.repo in #3595) and custom proxy CA cert support (introduced in #4870) do not extend to this path because the system git binary cannot receive proxy configuration programmatically.

Scope

Once PR #4841 (replace go-getter's GitGetter shellout with go-git) lands, the following can be addressed for helm.chart git downloads in the fleet apply main container:

1. SSH through HTTPS proxy — reuse pkg/git/proxy.go (implemented in [SURE-9994] Fleet support for SSH proxy servers #3595) via go-git's CloneOptions.
2. Custom proxy CA cert — deliver the proxy CA cert to the main container (currently only the init container receives it) and load it programmatically using x509.SystemCertPool() + append, consistent with Make CA bundle configurable for HTTPS_PROXY #4870.

Dependency

Blocked on #4841.

[p-se]

⚠️

gitDownload already supported SSH helm.chart URLs through an HTTPS proxy before this change. The CONNECT tunnel and its PROXY_CA_BUNDLE trust were wired up when #3595 introduced pkg/git/proxy.go, and the helm.chart code path started using gitDownload when #4841 replaced go-getter. The gap closed here is narrower: gitDownload was not merging PROXY_CA_BUNDLE into CloneOptions.CABundle, so git::https:// chart URLs cloned through an HTTPS proxy with a self-signed CA would fail TLS verification. This commit appends PROXY_CA_BUNDLE to auth.CABundle before passing the combined PEM to go-git, mirroring what newHTTPConnectDialer already does for the proxy connection itself.

The QA template covers both SSH and HTTPS helm.chart git URLs. The SSH test validates the end-to-end proxy path through gitDownload (previously only exercised via GitRepo.spec.repo in #4870's QA), while the HTTPS test directly targets the change introduced here.

[p-se]

QA Template – HTTPS Proxy with Custom CA for fleet.yaml helm.chart git URLs

Background

Fleet's fleet.yaml supports a helm.chart field that can reference a Helm chart stored in a git repository. After PR #4841 replaced go-getter's GitGetter with go-git, the fleet apply main container now clones those git URLs via bundlereader.gitDownload.

This change (issue #4869) extends the PROXY_CA_BUNDLE support introduced in #4870 to that path: gitDownload now reads PROXY_CA_BUNDLE from the environment and merges it into the CA bundle used for go-git TLS verification, and passes ProxyOptions from the environment to go-git so that SSH repos can be tunnelled through the HTTPS proxy.

In summary, this covers the path that was previously excluded in #4870's QA template:

go-getter SSH URLs in the fleet.yaml helm.chart field (e.g. git::ssh://git@github.com/...) are not yet supported and are out of scope for this QA pass.

Prerequisites

• A Kubernetes cluster with the Fleet release under test available (alpha/beta/RC).
• kubectl access to the cluster.
• Access to fleet-test-data (or a fork) via SSH — either a deploy key on the upstream repo or on your fork. The repo must contain a path with a valid Helm chart (e.g. helm/), referenced from a fleet.yaml.
• A self-signed CA certificate and the corresponding proxy certificate/key pair. The steps below show how to generate them.

Generate a Self-Signed Proxy CA and Certificate

Run the following on your workstation. The resulting files will be used both to configure the proxy container and to populate proxyCACert in Fleet's Helm values.

# CA key and certificate
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
  -subj "/CN=Fleet QA Proxy CA"

# Proxy server key and CSR
openssl genrsa -out proxy.key 4096
openssl req -new -key proxy.key -out proxy.csr \
  -subj "/CN=proxy.fleet-local.svc"

# Sign the proxy cert with our CA, including the SAN for the in-cluster hostname
openssl x509 -req -days 365 -in proxy.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out proxy.crt \
  -extfile <(printf "subjectAltName=DNS:proxy.fleet-local.svc,DNS:proxy.fleet-local.svc.cluster.local")

proxy.yaml

Generate proxy.yaml from the CA certificate produced in the previous step:

cat > proxy.yaml <<EOF
proxy: "https://fleet:fleet@proxy.fleet-local.svc:3129"
proxyCACert: |
$(sed 's/^/  /' ca.crt)
EOF

Install or upgrade Fleet using this file:

helm upgrade --install fleet-crd <repo>/fleet-crd --version <version> \
  -n cattle-fleet-system --create-namespace

helm upgrade --install fleet <repo>/fleet --version <version> \
  -n cattle-fleet-system \
  -f proxy.yaml

Setup

1. Create the proxy TLS Secret

kubectl create secret tls proxy-tls \
  --cert=proxy.crt \
  --key=proxy.key \
  -n fleet-local

2. Deploy the HTTPS Squid proxy

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxy
  namespace: fleet-local
  labels:
    app: proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proxy
  template:
    metadata:
      labels:
        app: proxy
    spec:
      containers:
        - name: proxy
          image: pseidensal/squid-proxy
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 3128
              protocol: TCP
            - name: https
              containerPort: 3129
              protocol: TCP
          volumeMounts:
            - name: tls
              mountPath: /etc/squid/proxy.crt
              subPath: tls.crt
              readOnly: true
            - name: tls
              mountPath: /etc/squid/proxy.key
              subPath: tls.key
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: proxy-tls
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: proxy
  namespace: fleet-local
  labels:
    app: proxy
spec:
  type: ClusterIP
  selector:
    app: proxy
  ports:
    - name: http
      port: 3128
      targetPort: 3128
      protocol: TCP
    - name: https
      port: 3129
      targetPort: 3129
      protocol: TCP
EOF

3. Create the SSH credential Secret

Replace ~/.ssh/id_ed25519 with the path to your deploy key:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitcredential
  namespace: fleet-local
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
$(sed 's/^/    /' ~/.ssh/id_ed25519)
EOF

4. Verify the PROXY_CA_BUNDLE env var is set in all relevant deployments

kubectl set env deployment/fleet-controller -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE
kubectl set env deployment/gitjob -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE
kubectl set env deployment/helmops -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE

All three should print the PROXY_CA_BUNDLE entry. The fleet apply Job pods spawned by gitjob inherit this from the gitjob deployment.

Test Cases

Test 1 – helm.chart SSH git URL cloned through HTTPS proxy

This tests that bundlereader.gitDownload picks up PROXY_CA_BUNDLE and routes through the proxy when cloning a Helm chart from a git SSH URL in fleet.yaml.

1. The branch issue-4869 in p-se/fleet-test-data already contains a ready-to-use fleet.yaml at proxy-helm-chart/fleet.yaml:

helm:
  chart: git@github.com:p-se/fleet-test-data.git//simple-chart/config-chart
  releaseName: test-helm-chart-proxy

This references the existing simple-chart/config-chart Helm chart in the same fork via an SSH git URL with a //-subdirectory specifier, so no additional test data needs to be created.

2. Apply a GitRepo pointing to that branch:

kubectl apply -f - <<'EOF'
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-helm-chart-ssh-https-proxy
  namespace: fleet-local
spec:
  repo: git@github.com:p-se/fleet-test-data.git
  branch: issue-4869
  pollingInterval: 10s
  clientSecretName: gitcredential  # SSH key for the gitcloner init container (clones spec.repo)
  helmSecretName: gitcredential    # SSH key for the fleet apply main container (clones helm.chart git URLs)
  paths:
    - proxy-helm-chart
EOF

3. Wait for the GitRepo to reach Ready state:

kubectl get gitrepo test-helm-chart-ssh-https-proxy -n fleet-local -w

Expected: The GitRepo reaches Ready. The fleet apply main container successfully cloned the helm.chart SSH git URL through the HTTPS proxy using the custom CA.

Test 2 – helm.chart HTTPS git URL cloned through HTTPS proxy

This is the primary test for the changes: gitDownload now merges PROXY_CA_BUNDLE
into CloneOptions.CABundle, which is the go-git TLS trust path for HTTPS clones.

Note: A plain https:// value in helm.chart is treated as an HTTP archive download
(httpdownload.go) and does not exercise gitDownload. The git:: forced-scheme prefix
is required to route an HTTPS URL through gitDownload.

1. Add a fleet.yaml at proxy-helm-chart-https/fleet.yaml in the issue-4869 branch of
   p-se/fleet-test-data:

helm:
  chart: git::https://github.com/p-se/fleet-test-data.git//simple-chart/config-chart
  releaseName: test-helm-chart-https-proxy

No helmSecretName is needed — the repo is public and HTTPS clones require no credentials.

2. Apply a GitRepo pointing to that path:

kubectl apply -f - <<'EOF'
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-helm-chart-https-proxy
  namespace: fleet-local
spec:
  repo: git@github.com:p-se/fleet-test-data.git
  branch: issue-4869
  pollingInterval: 10s
  clientSecretName: gitcredential
  paths:
    - proxy-helm-chart-https
EOF

3. Wait for the GitRepo to reach Ready state:

kubectl get gitrepo test-helm-chart-https-proxy -n fleet-local -w

Expected: The GitRepo reaches Ready. The fleet apply main container cloned the
git::https:// chart URL through the HTTPS proxy, trusting the proxy's custom CA via
PROXY_CA_BUNDLE.

Test 3 – Proxy unavailable (negative test)

1. Scale down the proxy deployment:

kubectl scale deployment proxy -n fleet-local --replicas=0

2. Delete and re-apply the GitRepo from Test 1 or Test 2.

Expected: The GitRepo fails to sync with a connection error, confirming that traffic was routed through the proxy rather than connecting directly to GitHub.

3. Restore the proxy before continuing:

kubectl scale deployment proxy -n fleet-local --replicas=1

Test 4 – Failure without the CA bundle (negative test)

1. Reinstall Fleet with the HTTPS proxy URL but without proxyCACert:

cat > proxy-no-ca.yaml <<'EOF'
proxy: "https://fleet:fleet@proxy.fleet-local.svc:3129"
EOF

helm upgrade --install fleet <repo>/fleet --version <version> \
  -n cattle-fleet-system \
  -f proxy-no-ca.yaml

2. Apply the GitRepo from Test 2 (HTTPS helm.chart URL).

3. Observe that the GitRepo fails to sync. Check the error:

kubectl get gitrepo test-helm-chart-https-proxy -n fleet-local -o yaml | grep -A2 message

Expected: The status shows a TLS-related error such as:
x509: certificate signed by unknown authority

This confirms that PROXY_CA_BUNDLE is required for git::https:// chart URLs through
an HTTPS proxy with a custom CA — the staged change is load-bearing.

Exclusions

• The GitRepo.spec.repo SSH cloning path (init container) was covered by #4870; this QA pass focuses on the fleet apply main container path only.
• Plain https:// values in helm.chart are routed as HTTP archive downloads (httpdownload.go) and do not exercise gitDownload; only git::https:// URLs are in scope here.
• Publicly trusted proxy certificates (no proxyCACert needed) are out of scope; that scenario worked before this change.