App Framework: Allow alternative CA authorities for S3 buckets

[gjanders]

Please select the type of request

Bug

Tell us more

Describe the request

2023-03-06T09:46:18.392485856Z  ERROR   GetAppListFromRemoteBucket      Unable to get apps list {"controller": "clustermanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "ClusterManager", "C
lusterManager": {"name":"mobiles-cm","namespace":"mobiles"}, "namespace": "mobiles", "name": "mobiles-cm", "reconcileID": "517c5a41-f927-4e6c-9d11-4bd9a72a9753", "name": "mobiles-cm", "namespace": "mobiles", "ap
pSource": "clusterApps", "error": "got an object error: Get \"https://10.x.x.x/k8s_mobile-idx-config/?location=\": x509: certificate signed by unknown authority for bucket: k8s_mobile-iot-idx-config"}

Expected behavior
The S3 bucket should be allowed even if a company CA certificate i used on-prem to sign the server (which is common for on-prem object storage).

Splunk setup on K8S
Splunk operator 2.2.0

Reproduction/Testing steps

spec:
  appRepo:
    appInstallPeriodSeconds: 90
    appSources:
    - location: clusterApps/
      name: clusterApps
    appsRepoPollIntervalSeconds: 900
    defaults:
      scope: cluster
      volumeName: volume_app_repo_us
    installMaxRetries: 2
    volumes:
    - endpoint: https://10.x.x.x
      name: volume_app_repo_us
      path: k8s_mobile-iot-idx-config/
      provider: minio
      region: us-west-2
      secretRef: mobile-iot-s3-secret
      storageType: s3

K8s environment

k8s cluster

Proposed changes(optional)

Allow self-signed/CA signed certificates from s3 storage provider

K8s collector data(optional)
Let me know if you need this

Additional context(optional)
If there is a flag or switch I can use to disable the validation please let me know

[vivekr-splunk]

@gjanders we are looking into this issue. do you want us to ignore the certificate validation? let me check if we can set it up as configurable value

[gjanders]

I think that ignoring certificate validation may be the easiest option and I'm happy to use that.
An alternative option may to be add corporate CA certificates into the system but I'm happy to disable certificate validation for now....

Thankyou!

[gjanders]

I know this issue was logged very recently, however do you have a timeline or a workaround in the near future?

If not I will have to temporarily avoid the app framework to keep the project moving forward.

Thanks

[vivekr-splunk]

@gjanders I have a solution in my mind. but I need sometime to test it out. even if the solution works we will only be releasing in May first week. If you want to create custom build from code you can always do that just change config in pkg/splunk/client/awss3client.go

config := &aws.Config{
		Region:                        aws.String(region),
		MaxRetries:                    aws.Int(3),
		HTTPClient:                    &httpClient,
		DisableSSL:                    aws.Bool(true),
		LogLevel:                      aws.LogLevel(aws.LogDebug | aws.LogDebugWithHTTPBody | aws.LogDebugWithRequestErrors | aws.LogDebugWithRequestErrors | aws.LogDebugWithSigning),
		Logger:                        aws.NewDefaultLogger(),
		CredentialsChainVerboseErrors: aws.Bool(true),
	}

remember this is not tested but I am hoping this will work.

[gjanders]

Thanks, I'll try to see if I can test this in the near future...

[gjanders]

I forgot to mention that I had switched to the minio provider as using the AWS provider my path ended up on the AWS URL even when I had the endpoint pointing to a local IP address:
k8s-config-testing.s3.ap-southeast-2.amazonaws.com

With endpoint: https://10....

Do you have a code snippet for minio I can test?

[gjanders]

I found that I could potentially modify transport.go for minio, on line 72 I added:
InsecureSkipVerify: true,
However I couldn't determine how to re-package this back into the operator build so after spending some time on it I gave up.
I also have attempted to modify the AWS client to use the endpoint I specify but so far without success