Operator normalizes single-element grant arrays to strings, causing Helm upgrade conflicts

[kalavt]

The ClickHouse Operator appears to normalize single-element arrays in the spec.configuration.users.<username>/grants/query field to string values during reconciliation. This causes Helm upgrade conflicts when later adding more grants to the array.

Environment

• ClickHouse Operator Version: 0.25.5
• Kubernetes Version: (your version)
• Helm Version: (your version)
• Deployment Method: Helm chart

Steps to Reproduce

1. Initial deployment with a single grant:

Deploy a CHI resource via Helm with one grant:

spec:
  configuration:
    users:
      metabase/grants/query:
        - "GRANT SELECT ON logs.*"

2. Check the stored resource:

kubectl get chi clickhouse -n observability -o yaml | grep -A 5 "metabase/grants"

Expected: Array format

metabase/grants/query:
  - "GRANT SELECT ON logs.*"

Actual: String format (after operator reconciliation)

metabase/grants/query: "GRANT SELECT ON logs.*"

3. Attempt to add more grants:

Update the Helm values to include multiple grants:

spec:
  configuration:
    users:
      metabase/grants/query:
        - "GRANT SELECT ON logs.*"
        - "GRANT SELECT ON system.processes"
        - "GRANT KILL QUERY ON *.*"

4. Run Helm upgrade:

helm upgrade clickhouse ./chart -n observability

Result: Helm reports a conflict error:

Error: UPGRADE FAILED: conflict occurred while applying object observability/clickhouse 
clickhouse.altinity.com/v1, Kind=ClickHouseInstallation: 
Apply failed with 1 conflict: conflict with "helm" using clickhouse.altinity.com/v1: 
.spec.configuration.users.metabase/grants/query

Root Cause Analysis

The issue occurs because:

1. The CRD uses x-kubernetes-preserve-unknown-fields: true for the users field
2. When a single-element array is deployed, the operator normalizes it to a string during reconciliation
3. When Helm tries to update from string to array, it detects a type incompatibility and reports a conflict
4. Helm's three-way strategic merge cannot automatically resolve type changes (string → array)

Impact

• Helm deployments fail when adding additional grants to users that initially had only one grant
• Requires manual intervention (kubectl patch) to resolve
• Inconsistent behavior: 2+ element arrays are preserved, but single-element arrays are converted
• Breaks GitOps workflows where configuration is managed declaratively through Helm

Workaround

Option 1: Always use 2+ grants (add a harmless grant):

users:
  - name: metabase
    grants:
      - "GRANT SELECT ON logs.*"
      - "GRANT SHOW ON *.*"  # Dummy grant to prevent normalization

Option 2: Use kubectl patch to update grants:

kubectl patch chi clickhouse -n observability --type=json -p='[...]'

Expected Behavior

The operator should preserve the array format even for single-element arrays, to maintain consistency and avoid Helm conflicts.

Suggested Solutions

1. Disable single-element array normalization for grant fields
2. Add a configuration option to control normalization behavior
3. Document this behavior prominently in the operator documentation
4. Preserve original data types from the API server without modification

Additional Context

This behavior is particularly problematic for teams using:

• Helm for infrastructure-as-code
• GitOps workflows (ArgoCD, Flux)
• Gradual permission expansion (starting with minimal grants)

The normalization might be intended for optimization, but it creates operational challenges in production environments where declarative configuration management is critical.

Related Code

The issue likely originates in the operator's reconciliation logic where it processes the users configuration and generates the ConfigMap for ClickHouse.