fix(storage): disable checksum validation when deploying managed storage (#272) (#273)

(cherry picked from commit 62558e8)

Co-authored-by: Andrew Azores <[email protected]>

Author: mergify[bot]

charts/cryostat/README.md

@@ -185,6 +185,7 @@ certificate issuance and rotation.
 | `storage.image.tag`                               | Tag for the storage container image                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `cryostat-v4.1`                     |
 | `storage.storageSecretName`                       | Name of the secret containing the object storage secret access key. This secret must contain a STORAGE_ACCESS_KEY secret which is the object storage secret access key. It must not be updated across chart upgrades, or else the connection between Cryostat components and object storage will not be able to initialize. If using an external S3 provider requiring authentication then this **must** be provided. It is recommended that the secret should be marked as immutable to avoid accidental changes to secret's data. More details: [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable)                                                                                                           | `""`                                |
 | `storage.provider.url`                            | URL to the S3 object storage provider instance. This can be an in-cluster self-hosted instance with a hostname like s3.storage.local, or it can be an external commercial service. This should include scheme, host, and port. User authenication information should be provided using a *Secret* and *storage.storageSecretName*. If this is not specified then a managed [cryostat-storage](https://github.com/cryostatio/cryostat-storage) instance will be automatically deployed and configured. If an unmanaged S3 instance is specified here then other storage configuration settings (such as at-rest encryption, Pod annotations, Service configurations) do not apply. Production installations of Cryostat should not rely on `cryostat-storage` | `""`                                |
+| `storage.provider.useChecksumValidation`          | whether PUT object request checksum validations are used. These should normally be enabled, but are known to cause issues with SeaweedFS/cryostat-storage and later S3 SDK versions. This is *true* by default when storage.provider.url is configured, but if not configured and cryostat-storage is deployed then this will be taken as *false*.                                                                                                                                                                                                                                                                                                                                                                                                           | `true`                              |
 | `storage.provider.usePathStyleAccess`             | whether path-style accesses are used for ex. object buckets. If path style access is not used then DNS subdomain resolution will be used. This is *true* by default for broader compatibility for low-footprint storage container installations, but subdomain resolution generally offers better performance if it is available and may be required for use with commercial storage providers.                                                                                                                                                                                                                                                                                                                                                              | `true`                              |
 | `storage.provider.usePresignedRecordingTransfers` | whether object storage presigned GET URLs should be used for transferring files between Cryostat components (ex. for automated analysis report generation). If this is disabled then Cryostat will act as a "network pipe" between other components and handle streaming file contents. This is *true* by default to reduce network utilization and request latency                                                                                                                                                                                                                                                                                                                                                                                          | `true`                              |
 | `storage.provider.usePresignedDownloads`          | whether object storage presigned GET URLs should be used for downloading files via the user's browser. If this is disabled then Cryostat will act as a "network pipe" between storage and the user's browser and handle streaming file contents. If the object storage URLs are not accessible from the user's network location then this must be disabled, otherwise enabling it will reduce network utilization and request latency. This is *false* by default                                                                                                                                                                                                                                                                                            | `false`                             |

charts/cryostat/templates/cryostat_deployment.yaml

@@ -132,6 +132,11 @@ spec:
             # if an external provider URL is supplied then a region must also be supplied.
             # Otherwise we are deploying a managed storage instance and can set a default value
             value: {{ ternary .Values.storage.provider.region "us-east-1" (not (empty .Values.storage.provider.url)) }}
+          - name: QUARKUS_S3_CHECKSUM_VALIDATION
+            # if an external provider URL is supplied then use the configuration checksum validation value.
+            # Otherwise we are deploying a managed storage instance and should override this to false, since it is known to cause issues.
+            # See https://github.com/cryostatio/cryostat/issues/948
+            value: {{ ternary (quote .Values.storage.provider.useChecksumValidation) (quote "false") (not (empty .Values.storage.provider.url)) }}
           - name: QUARKUS_S3_AWS_CREDENTIALS_TYPE
             value: {{ .Values.storage.provider.authentication.credentialsType }}
           - name: AWS_ACCESS_KEY_ID

charts/cryostat/tests/cryostat_deployment_test.yaml

@@ -127,6 +127,9 @@ tests:
       - equal:
           path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_ENDPOINT_OVERRIDE')].value
           value: "http://RELEASE-NAME-cryostat-storage:8333"
+      - equal:
+          path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_CHECKSUM_VALIDATION')].value
+          value: "false"
       - notExists:
           path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_SYNC_CLIENT_TLS_TRUST_MANAGERS_PROVIDER_TYPE')]
       - equal:
@@ -208,6 +211,7 @@ tests:
       storage:
         provider:
           url: 'https://s3.example.com:1234'
+          useChecksumValidation: true
           usePathStyleAccess: true
           usePresignedRecordingTransfers: false
           region: 'a-b1'
@@ -217,6 +221,9 @@ tests:
       - equal:
           path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_ENDPOINT_OVERRIDE')].value
           value: "https://s3.example.com:1234"
+      - equal:
+          path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_CHECKSUM_VALIDATION')].value
+          value: "true"
       - equal:
           path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_PATH_STYLE_ACCESS')].value
           value: "true"
@@ -233,6 +240,31 @@ tests:
           path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='STORAGE_METADATA_STORAGE_MODE')].value
           value: "bucket"
 
+  - it: should allow configuration of external object storage provider without checksum validation
+    set:
+      storage:
+        provider:
+          url: 'https://s3.example.com:6789'
+          useChecksumValidation: false
+    asserts:
+      - equal:
+          path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_ENDPOINT_OVERRIDE')].value
+          value: "https://s3.example.com:6789"
+      - equal:
+          path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_CHECKSUM_VALIDATION')].value
+          value: "false"
+
+  - it: should not allow configuration of managed object storage provider with checksum validation
+    set:
+      storage:
+        provider:
+          # no url configured, so a managed instance will be deployed
+          useChecksumValidation: true
+    asserts:
+      - equal:
+          path: spec.template.spec.containers[?(@.name=='cryostat')].env[?(@.name=='QUARKUS_S3_CHECKSUM_VALIDATION')].value
+          value: "false"
+
   - it: should allow configuration of presigned downloads separately from presigned transfers
     set:
       storage:

charts/cryostat/values.schema.json

@@ -681,6 +681,11 @@
                             "description": "URL to the S3 object storage provider instance. This can be an in-cluster self-hosted instance with a hostname like s3.storage.local, or it can be an external commercial service. This should include scheme, host, and port. User authenication information should be provided using a *Secret* and *storage.storageSecretName*. If this is not specified then a managed [cryostat-storage](https://github.com/cryostatio/cryostat-storage) instance will be automatically deployed and configured. If an unmanaged S3 instance is specified here then other storage configuration settings (such as at-rest encryption, Pod annotations, Service configurations) do not apply. Production installations of Cryostat should not rely on `cryostat-storage`",
                             "default": ""
                         },
+                        "useChecksumValidation": {
+                            "type": "boolean",
+                            "description": "whether PUT object request checksum validations are used. These should normally be enabled, but are known to cause issues with SeaweedFS/cryostat-storage and later S3 SDK versions. This is *true* by default when storage.provider.url is configured, but if not configured and cryostat-storage is deployed then this will be taken as *false*.",
+                            "default": true
+                        },
                         "usePathStyleAccess": {
                             "type": "boolean",
                             "description": "whether path-style accesses are used for ex. object buckets. If path style access is not used then DNS subdomain resolution will be used. This is *true* by default for broader compatibility for low-footprint storage container installations, but subdomain resolution generally offers better performance if it is available and may be required for use with commercial storage providers.",

charts/cryostat/values.yaml

@@ -280,6 +280,8 @@ storage:
   provider:
     ## @param storage.provider.url URL to the S3 object storage provider instance. This can be an in-cluster self-hosted instance with a hostname like s3.storage.local, or it can be an external commercial service. This should include scheme, host, and port. User authenication information should be provided using a *Secret* and *storage.storageSecretName*. If this is not specified then a managed [cryostat-storage](https://github.com/cryostatio/cryostat-storage) instance will be automatically deployed and configured. If an unmanaged S3 instance is specified here then other storage configuration settings (such as at-rest encryption, Pod annotations, Service configurations) do not apply. Production installations of Cryostat should not rely on `cryostat-storage`
     url: ""
+    ## @param storage.provider.useChecksumValidation whether PUT object request checksum validations are used. These should normally be enabled, but are known to cause issues with SeaweedFS/cryostat-storage and later S3 SDK versions. This is *true* by default when storage.provider.url is configured, but if not configured and cryostat-storage is deployed then this will be taken as *false*.
+    useChecksumValidation: true
     ## @param storage.provider.usePathStyleAccess whether path-style accesses are used for ex. object buckets. If path style access is not used then DNS subdomain resolution will be used. This is *true* by default for broader compatibility for low-footprint storage container installations, but subdomain resolution generally offers better performance if it is available and may be required for use with commercial storage providers.
     usePathStyleAccess: true
     ## @param storage.provider.usePresignedRecordingTransfers whether object storage presigned GET URLs should be used for transferring files between Cryostat components (ex. for automated analysis report generation). If this is disabled then Cryostat will act as a "network pipe" between other components and handle streaming file contents. This is *true* by default to reduce network utilization and request latency