[abctl] local install fails: Helm patch Deployment error due to env var order mismatch ($setElementOrder)

[ayanguas]

What happened?

When running abctl local install to update my local Airbyte deployment (on a kind Kubernetes cluster), the installation fails during the Helm upgrade step. The process correctly detects the existing cluster and resources, but the Helm chart fails to patch the airbyte-abctl-workload-api-server Deployment.

The error is related to an environment variable patch list not matching the expected $setElementOrder. The debug output also shows that some environment variables (e.g., AB_JWT_SIGNATURE_SECRET) appear twice in the patch list.

Environment: I'm running the upgrade process on an EC2 instance of type m6a.2xlarge.

Steps to reproduce the problem:

1. Set up a local Airbyte cluster previously using abctl local init and abctl local deploy.
2. Run abctl local install to upgrade to the latest version (v1.6.2).
3. Wait for Helm to apply the chart.
4. Observe that the process fails when patching airbyte-abctl-workload-api-server.

What did you expect to happen?

I expected abctl local install to successfully upgrade the local Airbyte deployment to the newest version without errors, properly applying the Helm chart and updating all relevant Kubernetes resources — including the airbyte-abctl-workload-api-server Deployment — without encountering conflicts or patch ordering issues.

Abctl Version

Details

$ abctl version
(version: v0.25.0)

Docker Version

Details

$ docker version
Client:
 Version:           25.0.5
 API version:       1.44
 Go version:        go1.22.5
 Git commit:        5dc9bcc
 Built:             Wed Aug 21 00:00:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          25.0.6
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.22.5
  Git commit:       b08a51f
  Built:            Wed Aug 21 00:00:00 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        2c9f5602f0ba3d9da1c2596322dfc4e156844890
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS Version

Details

# On Linux:
$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20241010"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2028-03-15"

[ayanguas]

Log:

DEBUG helm: Patch Deployment "airbyte-abctl-cron" in namespace airbyte-abctl DEBUG helm: Patch Deployment "airbyte-abctl-server" in namespace airbyte-abctl DEBUG helm: Patch Deployment "airbyte-abctl-temporal" in namespace airbyte-abctl DEBUG helm: Patch Deployment "airbyte-abctl-webapp" in namespace airbyte-abctl DEBUG helm: Patch Deployment "airbyte-abctl-worker" in namespace airbyte-abctl DEBUG helm: Patch Deployment "airbyte-abctl-workload-api-server" in namespace airbyte-abctl DEBUG helm: error updating the resource "airbyte-abctl-workload-api-server": cannot patch "airbyte-abctl-workload-api-server" with kind Deployment: The order in patch list: [map[name:AIRBYTE_EDITION valueFrom:map[configMapKeyRef:map[key:AIRBYTE_EDITION name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STEP valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STEP name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_OTLP_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_OTLP_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:OTEL_COLLECTOR_ENDPOINT valueFrom:map[configMapKeyRef:map[key:OTEL_COLLECTOR_ENDPOINT name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STATSD_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STATSD_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:SEGMENT_WRITE_KEY valueFrom:map[configMapKeyRef:map[key:SEGMENT_WRITE_KEY name:airbyte-abctl-airbyte-env]]] map[name:TRACKING_STRATEGY valueFrom:map[configMapKeyRef:map[key:TRACKING_STRATEGY name:airbyte-abctl-airbyte-env]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:MICRONAUT_ENVIRONMENTS valueFrom:map[configMapKeyRef:map[key:WORKERS_MICRONAUT_ENVIRONMENTS name:airbyte-abctl-airbyte-env]]] map[name:AIRBYTE_API_AUTH_HEADER_NAME value:X-Airbyte-Auth] map[name:AIRBYTE_API_AUTH_HEADER_VALUE value:Internal worker] map[name:API_AUTHORIZATION_ENABLED value:true] map[name:DATABASE_USER valueFrom:map[secretKeyRef:map[key:DATABASE_USER name:airbyte-abctl-airbyte-secrets]]] map[name:DATABASE_PASSWORD valueFrom:map[secretKeyRef:map[key:DATABASE_PASSWORD name:airbyte-abctl-airbyte-secrets]]]] doesn't match $setElementOrder list: [map[name:AIRBYTE_VERSION] map[name:AIRBYTE_EDITION] map[name:INTERNAL_API_HOST] map[name:AIRBYTE_API_HOST] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STEP] map[name:MICROMETER_METRICS_OTLP_ENABLED] map[name:OTEL_COLLECTOR_ENDPOINT] map[name:MICROMETER_METRICS_STATSD_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TEMPORAL_HOST] map[name:TRACKING_STRATEGY] map[name:WORKLOAD_API_BEARER_TOKEN] map[name:AB_JWT_SIGNATURE_SECRET] map[name:MICRONAUT_ENVIRONMENTS] map[name:AB_JWT_SIGNATURE_SECRET] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:API_AUTHORIZATION_ENABLED] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:AIRBYTE_INSTALLATION_ID]] DEBUG helm: Patch Deployment "airbyte-abctl-workload-launcher" in namespace airbyte-abctl DEBUG helm: warning: Upgrade "airbyte-abctl" failed: cannot patch "airbyte-abctl-workload-api-server" with kind Deployment: The order in patch list: [map[name:AIRBYTE_EDITION valueFrom:map[configMapKeyRef:map[key:AIRBYTE_EDITION name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STEP valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STEP name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_OTLP_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_OTLP_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:OTEL_COLLECTOR_ENDPOINT valueFrom:map[configMapKeyRef:map[key:OTEL_COLLECTOR_ENDPOINT name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STATSD_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STATSD_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:SEGMENT_WRITE_KEY valueFrom:map[configMapKeyRef:map[key:SEGMENT_WRITE_KEY name:airbyte-abctl-airbyte-env]]] map[name:TRACKING_STRATEGY valueFrom:map[configMapKeyRef:map[key:TRACKING_STRATEGY name:airbyte-abctl-airbyte-env]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:MICRONAUT_ENVIRONMENTS valueFrom:map[configMapKeyRef:map[key:WORKERS_MICRONAUT_ENVIRONMENTS name:airbyte-abctl-airbyte-env]]] map[name:AIRBYTE_API_AUTH_HEADER_NAME value:X-Airbyte-Auth] map[name:AIRBYTE_API_AUTH_HEADER_VALUE value:Internal worker] map[name:API_AUTHORIZATION_ENABLED value:true] map[name:DATABASE_USER valueFrom:map[secretKeyRef:map[key:DATABASE_USER name:airbyte-abctl-airbyte-secrets]]] map[name:DATABASE_PASSWORD valueFrom:map[secretKeyRef:map[key:DATABASE_PASSWORD name:airbyte-abctl-airbyte-secrets]]]] doesn't match $setElementOrder list: [map[name:AIRBYTE_VERSION] map[name:AIRBYTE_EDITION] map[name:INTERNAL_API_HOST] map[name:AIRBYTE_API_HOST] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STEP] map[name:MICROMETER_METRICS_OTLP_ENABLED] map[name:OTEL_COLLECTOR_ENDPOINT] map[name:MICROMETER_METRICS_STATSD_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TEMPORAL_HOST] map[name:TRACKING_STRATEGY] map[name:WORKLOAD_API_BEARER_TOKEN] map[name:AB_JWT_SIGNATURE_SECRET] map[name:MICRONAUT_ENVIRONMENTS] map[name:AB_JWT_SIGNATURE_SECRET] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:API_AUTHORIZATION_ENABLED] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:AIRBYTE_INSTALLATION_ID]] DEBUG helm: release upgrade failed: cannot patch "airbyte-abctl-workload-api-server" with kind Deployment: The order in patch list: [map[name:AIRBYTE_EDITION valueFrom:map[configMapKeyRef:map[key:AIRBYTE_EDITION name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STEP valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STEP name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_OTLP_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_OTLP_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:OTEL_COLLECTOR_ENDPOINT valueFrom:map[configMapKeyRef:map[key:OTEL_COLLECTOR_ENDPOINT name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STATSD_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STATSD_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:SEGMENT_WRITE_KEY valueFrom:map[configMapKeyRef:map[key:SEGMENT_WRITE_KEY name:airbyte-abctl-airbyte-env]]] map[name:TRACKING_STRATEGY valueFrom:map[configMapKeyRef:map[key:TRACKING_STRATEGY name:airbyte-abctl-airbyte-env]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:MICRONAUT_ENVIRONMENTS valueFrom:map[configMapKeyRef:map[key:WORKERS_MICRONAUT_ENVIRONMENTS name:airbyte-abctl-airbyte-env]]] map[name:AIRBYTE_API_AUTH_HEADER_NAME value:X-Airbyte-Auth] map[name:AIRBYTE_API_AUTH_HEADER_VALUE value:Internal worker] map[name:API_AUTHORIZATION_ENABLED value:true] map[name:DATABASE_USER valueFrom:map[secretKeyRef:map[key:DATABASE_USER name:airbyte-abctl-airbyte-secrets]]] map[name:DATABASE_PASSWORD valueFrom:map[secretKeyRef:map[key:DATABASE_PASSWORD name:airbyte-abctl-airbyte-secrets]]]] doesn't match $setElementOrder list: [map[name:AIRBYTE_VERSION] map[name:AIRBYTE_EDITION] map[name:INTERNAL_API_HOST] map[name:AIRBYTE_API_HOST] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STEP] map[name:MICROMETER_METRICS_OTLP_ENABLED] map[name:OTEL_COLLECTOR_ENDPOINT] map[name:MICROMETER_METRICS_STATSD_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TEMPORAL_HOST] map[name:TRACKING_STRATEGY] map[name:WORKLOAD_API_BEARER_TOKEN] map[name:AB_JWT_SIGNATURE_SECRET] map[name:MICRONAUT_ENVIRONMENTS] map[name:AB_JWT_SIGNATURE_SECRET] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:API_AUTHORIZATION_ENABLED] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:AIRBYTE_INSTALLATION_ID]] ERROR Failed to install airbyte/airbyte Helm Chart DEBUG looking at airbyte-abctl-airbyte-bootloader Succeeded() DEBUG found logs: Unsetting empty environment variable 'AB_INSTANCE_ DEBUG looking at airbyte-abctl-connector-builder-server-565b758c47-p4lbn Running() DEBUG found logs: Unsetting empty environment variable 'OTEL_COLLECT DEBUG looking at airbyte-abctl-cron-748ff66b87-n7j6w Running() DEBUG found logs: Unsetting empty environment variable 'OTEL_COLLECT DEBUG looking at airbyte-abctl-pod-sweeper-pod-sweeper-7cbbf9cf6d-49fxp Running() DEBUG found logs: Starting pod sweeper cycle: Will sweep succeeded p DEBUG looking at airbyte-abctl-server-6d8f5cd4c9-wjrdn Running() DEBUG found logs: Unsetting empty environment variable 'AIRBYTE_URL' DEBUG looking at airbyte-abctl-temporal-86df98d655-vfwt7 Running() DEBUG found logs: TEMPORAL_ADDRESS is not set, setting it to 10.244. DEBUG looking at airbyte-abctl-webapp-9686f9bd-mrs6h Running() DEBUG found logs: /docker-entrypoint.sh: /docker-entrypoint.d/ is no DEBUG looking at airbyte-abctl-worker-7797c448c7-h26tp Running() DEBUG found logs: Unsetting empty environment variable 'ACTIVITY_MAX DEBUG looking at airbyte-abctl-workload-api-server-6b466f58fd-nrl66 Running() DEBUG found logs: Unsetting empty environment variable 'VERSION' 202 DEBUG looking at airbyte-abctl-workload-launcher-5587dfd696-nzbzt Running() DEBUG found logs: Unsetting empty environment variable 'RUNNING_TTL_ DEBUG looking at airbyte-db-0 Running() DEBUG found logs: PostgreSQL Database directory appears to contain DEBUG looking at airbyte-minio-0 Running() DEBUG found logs: MinIO Object Storage Server Copyright: 2015-2023 M ERROR Unable to install Airbyte locally DEBUG Received unexpected event: *v1.Status DEBUG Event watcher completed. ERROR unable to install airbyte chart: unable to install helm: cannot patch "airbyte-abctl-workload-api-server" with kind Deployment: The order in patch list: [map[name:AIRBYTE_EDITION valueFrom:map[configMapKeyRef:map[key:AIRBYTE_EDITION name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STEP valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STEP name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_OTLP_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_OTLP_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:OTEL_COLLECTOR_ENDPOINT valueFrom:map[configMapKeyRef:map[key:OTEL_COLLECTOR_ENDPOINT name:airbyte-abctl-airbyte-env]]] map[name:MICROMETER_METRICS_STATSD_ENABLED valueFrom:map[configMapKeyRef:map[key:MICROMETER_METRICS_STATSD_ENABLED name:airbyte-abctl-airbyte-env]]] map[name:SEGMENT_WRITE_KEY valueFrom:map[configMapKeyRef:map[key:SEGMENT_WRITE_KEY name:airbyte-abctl-airbyte-env]]] map[name:TRACKING_STRATEGY valueFrom:map[configMapKeyRef:map[key:TRACKING_STRATEGY name:airbyte-abctl-airbyte-env]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:AB_JWT_SIGNATURE_SECRET valueFrom:map[secretKeyRef:map[key:jwt-signature-secret name:airbyte-auth-secrets]]] map[name:MICRONAUT_ENVIRONMENTS valueFrom:map[configMapKeyRef:map[key:WORKERS_MICRONAUT_ENVIRONMENTS name:airbyte-abctl-airbyte-env]]] map[name:AIRBYTE_API_AUTH_HEADER_NAME value:X-Airbyte-Auth] map[name:AIRBYTE_API_AUTH_HEADER_VALUE value:Internal worker] map[name:API_AUTHORIZATION_ENABLED value:true] map[name:DATABASE_USER valueFrom:map[secretKeyRef:map[key:DATABASE_USER name:airbyte-abctl-airbyte-secrets]]] map[name:DATABASE_PASSWORD valueFrom:map[secretKeyRef:map[key:DATABASE_PASSWORD name:airbyte-abctl-airbyte-secrets]]]] doesn't match $setElementOrder list: [map[name:AIRBYTE_VERSION] map[name:AIRBYTE_EDITION] map[name:INTERNAL_API_HOST] map[name:AIRBYTE_API_HOST] map[name:MICROMETER_METRICS_ENABLED] map[name:MICROMETER_METRICS_STEP] map[name:MICROMETER_METRICS_OTLP_ENABLED] map[name:OTEL_COLLECTOR_ENDPOINT] map[name:MICROMETER_METRICS_STATSD_ENABLED] map[name:MICROMETER_METRICS_STATSD_FLAVOR] map[name:SEGMENT_WRITE_KEY] map[name:STATSD_HOST] map[name:STATSD_PORT] map[name:TEMPORAL_HOST] map[name:TRACKING_STRATEGY] map[name:WORKLOAD_API_BEARER_TOKEN] map[name:AB_JWT_SIGNATURE_SECRET] map[name:MICRONAUT_ENVIRONMENTS] map[name:AB_JWT_SIGNATURE_SECRET] map[name:AIRBYTE_API_AUTH_HEADER_NAME] map[name:AIRBYTE_API_AUTH_HEADER_VALUE] map[name:API_AUTHORIZATION_ENABLED] map[name:DATABASE_HOST] map[name:DATABASE_PORT] map[name:DATABASE_DB] map[name:DATABASE_USER] map[name:DATABASE_PASSWORD] map[name:DATABASE_URL] map[name:AIRBYTE_INSTALLATION_ID]]

[qugu]

This is pretty much this issue: airbytehq/airbyte-platform#402