Route buckets per backend + verify multi-backend functional test end-to-end

The multi-backend functional test configured s3 + azureblob + filesystem but
only ever exercised one backend: the chart had no way to route a bucket to a
specific backend, so S3Proxy served every bucket from the first (default)
backend. The leg passed without touching azureblob or s3 — a false green.

Chart:
- Per-backend bucket-locator. Move s3proxy.bucket-locator.N out of the shared
  config block (which copied one global list into every backend file, so
  S3Proxy's first-file-wins de-dup meant only the first backend honored it) into
  each backend's own properties, driven by config.backends.<name>.bucketLocators.
- Per-backend credentials. Replace the single shared secret.properties (which
  emitted only the first-enabled backend's jclouds.credential and appended it to
  every backend file) with secret-common.properties + per-backend
  secret-<name>.properties; the merge initContainer appends common + the matching
  backend secret, so each backend gets only its own credential. Also fixes a
  latent GCS double-"jclouds.credential=" prefix.
- azureblob: emit the provider-agnostic jclouds.endpoint (was the
  jclouds.azureblob.endpoint typo) and render the computed default endpoint when
  only account is set.

Test:
- ci/functional/values/multi-backend.yaml routes smoke-s3-*/smoke-az-*/smoke-fs-*
  to their backends; azureblob uses the azureblob-sdk provider, which signs
  correctly against Azurite (the legacy azureblob provider mis-signs -> 403).
- ci/functional/assert-routing.sh writes one bucket per backend through s3proxy
  and verifies each object physically lands on its intended backend (MinIO for
  s3, Azurite for azureblob via az CLI, filesystem by elimination) and that no
  bucket leaks into another backend.
- functional-test.yaml: multi-backend and azureblob legs are now hard gates
  (all soft_fail removed).

No Chart.yaml version bump (handled separately), so verify-version fails by
design for now.

Verified on kind: multi-backend routing exclusive (s3->MinIO, az->Azurite,
fs->filesystem); azureblob-only, s3-only, filesystem-only all pass.

Author: CRThaze

.github/workflows/functional-test.yaml

@@ -5,27 +5,20 @@ name: Functional Test (kind)
 # s3proxy. Public repo → GitHub-hosted runner + kind, no cluster credentials, so
 # it is safe to run on every PR (including forks).
 #
-# Backend coverage:
-#   s3                      -> MinIO mock, HARD GATE (full S3 round-trip).
-#   filesystem / transient  -> HARD GATE (install + round-trip). The chart now
-#                              emits placeholder jclouds.identity/jclouds.credential
-#                              for local backends (DND-1442), so s3proxy starts and
-#                              serves. These legs guard that fix against regression.
-#   azureblob               -> Azurite mock, NON-BLOCKING (soft_fail) until DND-1416
-#                              fixes the endpoint property (jclouds.azureblob.endpoint
-#                              -> jclouds.endpoint); the failing round-trip is the
-#                              functional test demonstrably catching that regression.
+# Backend coverage (all HARD GATES):
+#   s3                      -> MinIO mock, full S3 round-trip.
+#   filesystem / transient  -> install + round-trip. The chart emits placeholder
+#                              jclouds.identity/jclouds.credential for local
+#                              backends, so s3proxy starts and serves.
+#   azureblob               -> Azurite mock, full round-trip through the fixed
+#                              jclouds.endpoint (provider-agnostic).
 #   multi-backend           -> s3 (MinIO) + azureblob (Azurite) + filesystem (PVC)
-#                              in ONE release, round-tripping one bucket per backend
-#                              (smoke-s3-*/smoke-az-*/smoke-fs-*). NON-BLOCKING:
-#                              cannot pass on current main — blocked by DND-1442 and
-#                              DND-1416, plus two multi-backend chart gaps (secret.yaml
-#                              emits only the first enabled backend's jclouds.credential
-#                              into the shared secret.properties, and there is no
-#                              per-backend bucket-locator to route buckets between
-#                              backends on the shared endpoint). See the notes in
-#                              ci/functional/values/multi-backend.yaml.
-# Each soft_fail leg auto-goes-green once its bug is fixed.
+#                              in ONE release, with per-backend bucket-locator
+#                              routing. assert-routing.sh writes one bucket per
+#                              backend through s3proxy and verifies each object
+#                              physically lands on its intended backend (MinIO for
+#                              s3, Azurite for azureblob, filesystem by elimination)
+#                              and that no bucket leaks into another backend.
 # gcs/b2/openstack-swift/rackspace are render-only (covered by lint-render + helm-diff):
 # gcs has no jclouds.endpoint override in the chart, and the others have no
 # lightweight in-cluster emulator.
@@ -48,7 +41,6 @@ jobs:
   functional:
     name: functional (${{ matrix.backend }})
     runs-on: ubuntu-latest
-    continue-on-error: ${{ matrix.soft_fail == true }}
     strategy:
       fail-fast: false
       matrix:
@@ -63,12 +55,10 @@ jobs:
           - backend: azureblob
             values: ci/functional/values/azureblob.yaml
             mock: ci/functional/mocks/azurite.yaml
-            soft_fail: true
           - backend: multi-backend
             values: ci/functional/values/multi-backend.yaml
             mock: ci/functional/mocks/minio.yaml ci/functional/mocks/azurite.yaml
-            buckets: smoke-s3 smoke-az smoke-fs
-            soft_fail: true
+            verify: routing
     env:
       NAMESPACE: s3proxy-test
       RELEASE: s3proxy
@@ -105,15 +95,19 @@ jobs:
             --values "${{ matrix.values }}" \
             --wait --timeout 5m
 
-      - name: Smoke test (S3 round-trip)
-        # matrix.buckets is a space-separated list of bucket prefixes — one
-        # round-trip per prefix (multi-backend scenarios use one per backend).
+      - name: Verify (${{ matrix.verify || 'round-trip' }})
+        # Single-backend legs: one S3 round-trip through s3proxy.
+        # multi-backend leg (verify=routing): assert each bucket physically lands
+        # on its intended backend (needs aws + az; both are on ubuntu-latest).
         run: |
-          chmod +x ci/functional/smoke-test.sh
-          for prefix in ${{ matrix.buckets || 'smoke' }}
-          do
-            ci/functional/smoke-test.sh "${RELEASE}" "${NAMESPACE}" 9000 test-access-key test-secret-key "${prefix}"
-          done
+          if [ "${{ matrix.verify }}" = "routing" ]
+          then
+            chmod +x ci/functional/assert-routing.sh
+            ci/functional/assert-routing.sh "${RELEASE}" "${NAMESPACE}"
+          else
+            chmod +x ci/functional/smoke-test.sh
+            ci/functional/smoke-test.sh "${RELEASE}" "${NAMESPACE}" 9000 test-access-key test-secret-key
+          fi
 
       - name: Diagnostics on failure
         if: failure()

charts/s3proxy/README.md.gotmpl

@@ -257,17 +257,27 @@ config:
       another-bucket: "actual-bucket-name"
 ```
 
-### Bucket Locator
+### Bucket Locator (routing buckets to backends)
 
-Assign specific buckets to different backends:
+When more than one backend is enabled, assign buckets to a specific backend with
+that backend's own `bucketLocators` list. S3Proxy reads bucket-locators per
+backend, so each list is emitted only into that backend's properties file. Glob
+patterns are supported. A bucket that matches no backend's list falls through to
+the first-enabled backend (the default).
 
 ```yaml
 config:
-  buckets:
-    locator:
-      - "bucket1"
-      - "bucket2"
-      - "*.test"  # Glob patterns supported
+  backends:
+    s3:
+      enabled: true
+      # ... credentials ...
+      bucketLocators:
+        - "prod-*"
+        - "customer-data"
+    filesystem:
+      enabled: true
+      bucketLocators:
+        - "scratch-*"   # everything else also lands here (first-enabled default)
 ```
 
 ## Monitoring

charts/s3proxy/templates/configmap.yaml

@@ -32,12 +32,11 @@ s3proxy.alias.{{ $key }}={{ $value }}
     {{- end }}
   {{- end }}
 
-  {{- if .Values.config.buckets.locator }}
-# Bucket locator
-    {{- range $index, $bucket := .Values.config.buckets.locator }}
-s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
-    {{- end }}
-  {{- end }}
+  {{- /* Bucket locators are per-backend: each backend's properties file lists
+         the buckets it owns (see config.backends.<name>.bucketLocators). They are
+         intentionally NOT emitted here in the shared config, because S3Proxy
+         de-duplicates locators across --properties files (first file wins), so a
+         shared list would route every bucket to only the first backend. */}}
 
   {{- if .Values.config.middlewares.readOnly }}
 # Read-only middleware
@@ -83,6 +82,9 @@ data:
     # properties file, even for local backends where the values are unused.
     jclouds.identity={{ .Values.config.backends.filesystem.identity | default "local" }}
     jclouds.credential={{ .Values.config.backends.filesystem.credential | default "local" }}
+  {{- range $index, $bucket := .Values.config.backends.filesystem.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.transient.enabled }}
@@ -99,6 +101,9 @@ data:
     # properties file, even for local backends where the values are unused.
     jclouds.identity={{ .Values.config.backends.transient.identity | default "local" }}
     jclouds.credential={{ .Values.config.backends.transient.credential | default "local" }}
+  {{- range $index, $bucket := .Values.config.backends.transient.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.s3.enabled }}
@@ -123,6 +128,9 @@ data:
   {{- if .Values.config.backends.s3.secretAccessKey.value }}
     # Credential will be merged from the secret properties file
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.s3.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.azureblob.enabled }}
@@ -131,15 +139,18 @@ data:
 
     # Azure Blob backend configuration
     jclouds.provider={{ .Values.config.backends.azureblob.provider }}
-  {{- if .Values.config.backends.azureblob.endpoint }}
-    jclouds.azureblob.endpoint={{ .Values.config.backends.azureblob.endpoint | default (printf "https://%s.blob.core.windows.net" .Values.config.backends.azureblob.account) }}
+  {{- if or .Values.config.backends.azureblob.endpoint .Values.config.backends.azureblob.account }}
+    jclouds.endpoint={{ .Values.config.backends.azureblob.endpoint | default (printf "https://%s.blob.core.windows.net" .Values.config.backends.azureblob.account) }}
   {{- end }}
   {{- if .Values.config.backends.azureblob.account }}
     jclouds.identity={{ .Values.config.backends.azureblob.account }}
   {{- end }}
   {{- if or .Values.config.backends.azureblob.key.value .Values.config.backends.azureblob.sasToken.value }}
     # Credentials will be merged from the secret properties file
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.azureblob.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.googleCloudStorage.enabled }}
@@ -161,6 +172,9 @@ data:
     # Private key credential will be merged from the secret properties file
     # jclouds.credential will be provided by the secret
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.googleCloudStorage.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.b2.enabled }}
@@ -175,6 +189,9 @@ data:
   {{- if .Values.config.backends.b2.applicationKey.value }}
     # Credential will be merged from the secret properties file
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.b2.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.openstackSwift.enabled }}
@@ -195,6 +212,9 @@ data:
   {{- if .Values.config.backends.openstackSwift.password.value }}
     # Credential will be merged from the secret properties file
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.openstackSwift.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}
 
 {{- if .Values.config.backends.rackspaceCloudfiles.enabled }}
@@ -213,4 +233,7 @@ data:
   {{- if .Values.config.backends.rackspaceCloudfiles.apiKey.value }}
     # Credential will be merged from the secret properties file
   {{- end }}
+  {{- range $index, $bucket := .Values.config.backends.rackspaceCloudfiles.bucketLocators }}
+    s3proxy.bucket-locator.{{ add $index 1 }}={{ $bucket }}
+  {{- end }}
 {{- end }}

charts/s3proxy/templates/deployment.yaml

@@ -44,15 +44,18 @@ spec:
               set -e
               echo "Merging configuration files..."
 
-              # Check if secret properties file exists
-              SECRET_FILE="/secret/secret.properties"
+              # secret-common.properties holds the client-facing auth and is
+              # appended to every backend file. Each backend additionally gets
+              # its own secret-<backend>.properties (its jclouds.credential), so
+              # one backend's credential never leaks into another's file.
+              COMMON_SECRET="/secret/secret-common.properties"
 
               # Loop through all properties files in the config directory
               for config_file in /config/*.properties
               do
                 if [ -f "$config_file" ]
                 then
-                  # Get the filename
+                  # Get the filename, e.g. backend-s3.properties
                   filename=$(basename "$config_file")
                   output_file="/merged-config/$filename"
 
@@ -61,11 +64,22 @@ spec:
                   # Copy base config file to output
                   cp "$config_file" "$output_file"
 
-                  # If secret file exists, append its contents (overriding duplicates)
-                  if [ -f "$SECRET_FILE" ]
+                  # Append the shared client-auth secret, if present
+                  if [ -f "$COMMON_SECRET" ]
                   then
                     echo "" >> "$output_file"  # Add newline separator
-                    cat "$SECRET_FILE" >> "$output_file"
+                    cat "$COMMON_SECRET" >> "$output_file"
+                  fi
+
+                  # Append this backend's own secret: backend-<name>.properties
+                  # pairs with secret-<name>.properties
+                  backend="${filename#backend-}"
+                  backend="${backend%.properties}"
+                  backend_secret="/secret/secret-${backend}.properties"
+                  if [ -f "$backend_secret" ]
+                  then
+                    echo "" >> "$output_file"  # Add newline separator
+                    cat "$backend_secret" >> "$output_file"
                   fi
                 fi
               done

charts/s3proxy/templates/secret.yaml

@@ -6,48 +6,53 @@ metadata:
     {{- include "s3proxy.labels" . | nindent 4 }}
 type: Opaque
 stringData:
-  # Properties file containing sensitive configuration values
-  # This will be merged with the backend properties files by the init container
-  secret.properties: |
+  # Sensitive properties merged into the backend properties files by the
+  # merge-configs initContainer:
+  #   * secret-common.properties  -> appended to EVERY backend file (client auth).
+  #   * secret-<backend>.properties -> appended ONLY to that backend's file, so
+  #     each backend gets its own jclouds.credential and never another backend's.
+  # Local backends (filesystem/transient) need no secret file; their placeholder
+  # jclouds.identity/credential come from the ConfigMap.
 {{- if and .Values.config.auth.identity .Values.config.auth.secret }}
-    # S3Proxy authentication credentials (for clients connecting to s3proxy)
+  secret-common.properties: |
+    # S3Proxy client authentication credentials (shared by all backends)
     s3proxy.identity={{ .Values.config.auth.identity }}
     s3proxy.credential={{ .Values.config.auth.secret }}
 {{- end }}
-
-{{- if .Values.config.backends.s3.enabled }}
-  {{- if .Values.config.backends.s3.secretAccessKey.value }}
-    # S3 backend credentials
+{{- if and .Values.config.backends.s3.enabled .Values.config.backends.s3.secretAccessKey.value }}
+  secret-s3.properties: |
+    # S3 backend credential
     jclouds.credential={{ .Values.config.backends.s3.secretAccessKey.value }}
-  {{- end }}
-{{- else if .Values.config.backends.azureblob.enabled }}
-  {{- if or .Values.config.backends.azureblob.key.value .Values.config.backends.azureblob.sasToken.value }}
+{{- end }}
+{{- if .Values.config.backends.azureblob.enabled }}
+{{- if or .Values.config.backends.azureblob.key.value .Values.config.backends.azureblob.sasToken.value }}
+  secret-azureblob.properties: |
     # Azure Blob backend credentials
-  {{- end }}
-  {{- if .Values.config.backends.azureblob.key.value }}
+{{- if .Values.config.backends.azureblob.key.value }}
     jclouds.credential={{ .Values.config.backends.azureblob.key.value }}
-  {{- end }}
-  {{- if .Values.config.backends.azureblob.sasToken.value }}
+{{- end }}
+{{- if .Values.config.backends.azureblob.sasToken.value }}
     jclouds.azureblob.sas={{ .Values.config.backends.azureblob.sasToken.value }}
-  {{- end }}
-{{- else if .Values.config.backends.googleCloudStorage.enabled }}
-  {{- if .Values.config.backends.googleCloudStorage.privateKey.value }}
-    # Google Cloud Storage backend credentials (privateKey stored in secret)
-    jclouds.credential=jclouds.credential={{ .Values.config.backends.googleCloudStorage.privateKey.value | trim | replace "\n" "\\n\\" }}
-  {{- end }}
-{{- else if .Values.config.backends.b2.enabled }}
-  {{- if .Values.config.backends.b2.applicationKey.value }}
-    # Backblaze B2 backend credentials
+{{- end }}
+{{- end }}
+{{- end }}
+{{- if and .Values.config.backends.googleCloudStorage.enabled .Values.config.backends.googleCloudStorage.privateKey.value }}
+  secret-google-cloud-storage.properties: |
+    # Google Cloud Storage backend credential (PEM private key)
+    jclouds.credential={{ .Values.config.backends.googleCloudStorage.privateKey.value | trim | replace "\n" "\\n\\" }}
+{{- end }}
+{{- if and .Values.config.backends.b2.enabled .Values.config.backends.b2.applicationKey.value }}
+  secret-b2.properties: |
+    # Backblaze B2 backend credential
     jclouds.credential={{ .Values.config.backends.b2.applicationKey.value }}
-  {{- end }}
-{{- else if .Values.config.backends.openstackSwift.enabled }}
-  {{- if .Values.config.backends.openstackSwift.password.value }}
-    # OpenStack Swift backend credentials
+{{- end }}
+{{- if and .Values.config.backends.openstackSwift.enabled .Values.config.backends.openstackSwift.password.value }}
+  secret-openstack-swift.properties: |
+    # OpenStack Swift backend credential
     jclouds.credential={{ .Values.config.backends.openstackSwift.password.value }}
-  {{- end }}
-{{- else if .Values.config.backends.rackspaceCloudfiles.enabled }}
-  {{- if .Values.config.backends.rackspaceCloudfiles.apiKey.value }}
-    # Rackspace Cloud Files backend credentials
+{{- end }}
+{{- if and .Values.config.backends.rackspaceCloudfiles.enabled .Values.config.backends.rackspaceCloudfiles.apiKey.value }}
+  secret-rackspace-cloudfiles.properties: |
+    # Rackspace Cloud Files backend credential
     jclouds.credential={{ .Values.config.backends.rackspaceCloudfiles.apiKey.value }}
-  {{- end }}
 {{- end }}