Make CA bundle configurable for HTTPS_PROXY

[p-se]

Background

Fleet supports routing outbound traffic through an HTTP/HTTPS proxy via the standard HTTP_PROXY/HTTPS_PROXY environment variables. When the proxy itself uses a custom or private CA certificate (i.e. an HTTPS proxy with a self-signed or enterprise CA), Fleet's components need to trust that CA or all connections through the proxy will fail with a TLS verification error.

Currently there is no way to configure a custom CA cert for the proxy in Fleet.

Scope

The following components need to trust the proxy CA:

1. gitjob-controller — fetches the latest commit from the git repo (polling).
2. gitjob pod, init container (gitcloner) — clones GitRepo.spec.repo using go-git.
3. HelmOp controller — polls for the latest chart version via bundlereader.ChartVersion. The HTTP transport (transportForAuth) uses x509.SystemCertPool() as the base but has no mechanism to inject the proxy CA separately from auth.CABundle (which is the chart repo's own CA).

This also applies to cloning over SSH through an HTTPS proxy (introduced in #3595): the proxy CA cert must be trusted when establishing the CONNECT tunnel to the proxy before the SSH connection is made.

The fleet apply main container and helm.chart downloads via go-getter are tracked separately in #4869.

Notes

• SSL_CERT_FILE/SSL_CERT_DIR are not suitable for injecting an additional CA cert because they replace the system cert pool rather than appending to it, which would break TLS verification for public git hosts.
• The preferred approach is to pass the cert PEM via an environment variable and load it programmatically using x509.SystemCertPool() + append, then pass the combined pool to go-git's HTTP transport.

[p-se]

QA Template – HTTPS Proxy with Custom CA Certificate

Background

Fleet can be configured to route traffic through an HTTP proxy via the proxy Helm value. When the proxy URL uses https://, the proxy itself presents a TLS certificate. If that certificate is self-signed or issued by a private CA, Fleet previously had no way to trust it, causing all git cloning and Helm chart fetching to fail with a TLS error.

This change adds a proxyCACert Helm value. When set, Fleet inlines the PEM-encoded certificate directly in each Deployment's pod template and exposes it to all relevant components as the PROXY_CA_BUNDLE environment variable:

• the fleet-controller deployment
• the gitjob deployment
• git cloner init containers (via proxyEnvVars())
• the HelmOps controller deployment

The CA is appended to the system cert pool at runtime so that well-known public CAs remain trusted alongside the proxy CA.

Note that this does not yet cover go-getter SSH URLs (specified via the helm.chart field in fleet.yaml). Issue #4869 tracks extending this support to that path once PR #4841 (replacing go-getter's GitGetter with go-git) lands.

Prerequisites

• A Kubernetes cluster with the Fleet release under test available (alpha/beta/RC).
• kubectl access to the cluster.
• Access to fleet-test-data (or a fork of it) via SSH — either an SSH deploy key on the upstream repo or on your fork.
• A self-signed CA certificate and the corresponding proxy certificate/key pair. The steps below show how to generate them.

Generate a Self-Signed Proxy CA and Certificate

Run the following on your workstation. The resulting files will be used both to configure the proxy container and to populate proxyCACert in Fleet's Helm values.

# CA key and certificate
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
  -subj "/CN=Fleet QA Proxy CA"

# Proxy server key and CSR
openssl genrsa -out proxy.key 4096
openssl req -new -key proxy.key -out proxy.csr \
  -subj "/CN=proxy.fleet-local.svc"

# Sign the proxy cert with our CA, including the SAN for the in-cluster hostname
openssl x509 -req -days 365 -in proxy.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out proxy.crt \
  -extfile <(printf "subjectAltName=DNS:proxy.fleet-local.svc,DNS:proxy.fleet-local.svc.cluster.local")

proxy.yaml

Generate proxy.yaml from the CA certificate produced in the previous step:

cat > proxy.yaml <<EOF
proxy: "https://fleet:fleet@proxy.fleet-local.svc:3129"
proxyCACert: |
$(sed 's/^/  /' ca.crt)
EOF

Install or upgrade Fleet using this file:

helm upgrade --install fleet-crd <repo>/fleet-crd --version <version> \
  -n cattle-fleet-system --create-namespace

helm upgrade --install fleet <repo>/fleet --version <version> \
  -n cattle-fleet-system \
  -f proxy.yaml

Setup

1. Create the proxy TLS Secret

Store the proxy certificate and key so the proxy container can serve HTTPS:

kubectl create secret tls proxy-tls \
  --cert=proxy.crt \
  --key=proxy.key \
  -n fleet-local

2. Deploy the HTTPS Squid proxy

Deploy a Squid proxy configured for HTTPS (port 3129) in the fleet-local namespace:

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxy
  namespace: fleet-local
  labels:
    app: proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: proxy
  template:
    metadata:
      labels:
        app: proxy
    spec:
      containers:
        - name: proxy
          image: pseidensal/squid-proxy
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 3128
              protocol: TCP
            - name: https
              containerPort: 3129
              protocol: TCP
          volumeMounts:
            - name: tls
              mountPath: /etc/squid/proxy.crt
              subPath: tls.crt
              readOnly: true
            - name: tls
              mountPath: /etc/squid/proxy.key
              subPath: tls.key
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: proxy-tls
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: proxy
  namespace: fleet-local
  labels:
    app: proxy
spec:
  type: ClusterIP
  selector:
    app: proxy
  ports:
    - name: http
      port: 3128
      targetPort: 3128
      protocol: TCP
    - name: https
      port: 3129
      targetPort: 3129
      protocol: TCP
EOF

3. Create the SSH credential Secret

Replace ~/.ssh/id_ed25519 with the path to your deploy key:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitcredential
  namespace: fleet-local
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
$(sed 's/^/    /' ~/.ssh/id_ed25519)
EOF

4. Verify Fleet with the HTTPS proxy and CA bundle

Install Fleet using the proxy.yaml from the proxy.yaml section above.

Verify that the fleet-controller, gitjob, and helmops deployments have the PROXY_CA_BUNDLE environment variable set:

kubectl set env deployment/fleet-controller -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE
kubectl set env deployment/gitjob -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE
kubectl set env deployment/helmops -n cattle-fleet-system --list | grep PROXY_CA_BUNDLE

Test Cases

Test 1 – GitRepo cloning via SSH through an HTTPS proxy

1. Apply the following GitRepo, substituting your fork's SSH URL:

kubectl apply -f - <<'EOF'
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-ssh-https-proxy
  namespace: fleet-local
spec:
  repo: git@github.com:p-se/fleet-test-data.git
  branch: master
  pollingInterval: 10s
  clientSecretName: gitcredential
  paths:
    - simple
EOF

2. Wait for the GitRepo to reach Ready state:

kubectl get gitrepo test-ssh-https-proxy -n fleet-local -w

Test 2 – Commit polling via HTTPS through an HTTPS proxy

1. Apply a GitRepo pointing to an HTTPS URL (public repository is fine):

kubectl apply -f - <<'EOF'
apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: test-https-https-proxy
  namespace: fleet-local
spec:
  repo: https://github.com/p-se/fleet-test-data.git
  branch: master
  pollingInterval: 10s
  paths:
    - simple
EOF

2. Wait for Ready state and confirm resources are deployed.

Test 3 – HelmOp fetching a chart through an HTTPS proxy

1. Apply a HelmOp that references a chart hosted on an HTTPS server reachable only through the proxy:

kubectl apply -f - <<'EOF'
apiVersion: fleet.cattle.io/v1alpha1
kind: HelmOp
metadata:
  name: test-helmop-https-proxy
  namespace: fleet-local
spec:
  helm:
    repo: https://stefanprodan.github.io/podinfo # lightweight test chart with HTTPS endpoint
    chart: podinfo
    version: "6.x"
EOF

2. Wait for the HelmOp to reach Ready state.

Test 4 – Failure without the CA bundle (regression / negative test)

This test confirms that the error is meaningful when the CA is not configured.

1. Reinstall Fleet with the HTTPS proxy but without proxyCACert:

cat > proxy.yaml <<'EOF'
proxy: "https://fleet:fleet@proxy.fleet-local.svc:3129"
EOF

helm upgrade --install fleet <repo>/fleet --version <version> \
  -n cattle-fleet-system \
  -f proxy.yaml

2. Apply the same SSH GitRepo from Test 1.

3. Observe that the GitRepo fails to sync.

4. Check the error in the GitRepo status:

kubectl get gitrepo test-ssh-https-proxy -n fleet-local -o yaml | grep -A2 message

Expected result: The status contains a TLS-related error such as:
http connect proxy: TLS handshake with proxy ...: x509: certificate signed by unknown authority

Test 5 – Proxy removed (negative test)

1. Reinstall Fleet with the HTTPS proxy and CA bundle as in the Setup section.

2. Apply the SSH GitRepo from Test 1 and wait for Ready.

3. Delete the proxy Service:

kubectl delete service proxy -n fleet-local

4. Delete and re-apply the GitRepo.

Expected result: The GitRepo fails to sync and shows a connection error, confirming that traffic was indeed going through the proxy rather than connecting directly.

Exclusions

• Testing with a publicly trusted proxy certificate (no proxyCACert needed) is out of scope; that scenario worked before this change.
• go-getter SSH URLs in the fleet.yaml helm.chart field (e.g. git::ssh://git@github.com/...) are not yet supported and are out of scope for this QA pass. This is tracked in #4869, which is blocked on PR Remove go-getter from GetContent and switch Docker image to bci-busybox #4841 replacing go-getter's GitGetter with go-git.