App Framework: it is not working with private buckets.

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request
splunk-operator application can't be configured with custom apps, if they are stored in private buckets and where it is forbidden to use secret keys.

Expected behavior
If there is no possibility attach AWS policy to pod, inctance policy should be used instead.
everything should work with s3://bucket_name notation on native way.

Splunk setup on K8S
EKS 1.24
splunk-operator 2.1.1
splunk 9.0.3

Reproduction/Testing steps
try any of next configuration:

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "https://splunk-operator-<AccountID>.s3-accesspoint.eu-central-1.amazonaws.com/"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]
      }

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "s3://splunk-operators-bucket"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]
      }

in all ways it drops logs like:

2023-01-17T09:49:22.534992739Z  ERROR   GetAppsList     Unable to list items in bucket  {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"lm","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "lm", "reconcileID": "216e03a6-2944-4e04-b75f-2afaca181255", "AWS S3 Bucket": "splunk-apps", "error": "AccessDenied: Access Denied\n\tstatus code: 403, request id: VPP022WAHETK9CCC, host id: dx2l8TLjNcD+0NNOEI8devQo9TdPYvF3TZGhR5UGkAxVQ/H40TmbME03vM9zLDOTHhijTiX5TLL="}

in our setup we can't use public access for buckets at all, as well as we can't use secret keys as authentication.

[yaroslav-nakonechnikov]

also, helm chart doesn't allow to pass env variables to splunk-operator, so it is not easy to add env vars in manager container, like:

  AWS_STS_REGIONAL_ENDPOINTS:   regional
     AWS_DEFAULT_REGION:           eu-central-1
     AWS_REGION:                   eu-central-1
     AWS_ROLE_ARN:                 arn:aws:iam::AccountID:role/name
     AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
     ```

[yaroslav-nakonechnikov]

another log:

023-01-17T13:46:33.187388246Z  ERROR   GetAppsList     Unable to list items in bucket  {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"lm","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "lm", "reconcileID": "0ec6ec50-80e2-45bc-beec-9aebcced1403", "AWS S3 Bucket": "splunk-apps", "error": "BucketRegionError: incorrect region, the bucket is not in 'eu-central-1' region at endpoint '', bucket is in 'us-east-1' region\n\tstatus code: 301, request id: 9SHNKVVZDKFBS1PC, host id: cj9lItWg+Rh9qKP4Vw58rHiOHyjZHSvVtm3FKaVlOFu+pZJ/AbXYza+0+Y14haaIatH/h1nbK91kB7m3mlnjLA=="}

config:

"appRepo" = {
        "appsRepoPollIntervalSeconds" = "86400"
        "appSources" = [
          {
            "location"   = "config-explorer_1715.tgz"
            "volumeName" = "splunk-apps"
            "scope"      = "local"
            "name"       = "Config Explorer"
          }
        ]
        "volumes" = [{
          "endpoint"    = "https://splunk-operator-AccountID.s3-accesspoint.eu-central-1.amazonaws.com/"
          "name"        = "splunk-apps"
          "path"        = "splunk-apps/"
          "provider"    = "aws"
          "storageType" = "s3"
          "region"      = "eu-central-1"
        }]

why it thinks that access point in different region?

[vivekr-splunk]

AWS IAM Service account is already supported in Splunk Operator. For Splunkd we are working add that feature. for splunk operator please follow the AWS documentation to add IAM service account to splunk-operator-controller-manager .
please follow the steps given here https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/

example are here

> eksctl utils associate-iam-oidc-provider --region=us-west-2 --cluster=vivek --approve
> eksctl create iamserviceaccount --name splunk-operator-controller-manager --namespace splunk-operator --cluster vivek --role-name "oidc-test-role"  --attach-policy-arn arn:aws:iam::667741767953:policy/oidc-test-policy --approve --override-existing-serviceaccounts

by adding the IAM service account, it automatically adds environment variables

AWS_ROLE_ARN:                 arn:aws:iam::AccountID:role/name
AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

let me know if you need any further clarifications.

[yaroslav-nakonechnikov]

splunk-operator - maybe. but it started to work only when i updated instance profile where splunk-operator was running.

anyway, still issues with using formats: aws allows at least 3 notations + accesspoints. Splunk operator supports only single way.

[vivekr-splunk]

@iaroslav-nakonechnikov we are working on supporting OIDC using AWS IAM serviceaccount for splunkd. will update you once it is supported.

[akondur]

Hi @iaroslav-nakonechnikov , please let us know if this issue of the instance profile has been resolved.

Could you elaborate on the different formats for AWS configurations with examples so that we can take a further look?