Opensearch as Search Engine Deployment with AWS IAM Authentication

[jz-xie]

Ask shown in the document here

Seems like only user and password is supported for Opensearch. It would be useful to have AWS IAM authentication supported as well.

[Prajwal214]

Ref - https://docs.open-metadata.org/latest/deployment/rds-iam-auth

[jz-xie]

Hi @Prajwal214

Does the IAM auth also works for OpenSearch?

[FFCMSouza]

My AWS Opensearch only have IAM and Cognito authentication. I was unable to connect with opensearch using either method.
Is there a way to authenticate Opensearch using any of these approachs?

[jzzzzzzzzz-z]

There is tool might be useful for you:

https://github.com/awslabs/aws-sigv4-proxy

You can try deploy this tool as a sidecar container of OpenMetadata to sign its OpenSearch requests, with you AWS IAM role attached in service account.

I have not deployed it myself because the latest version has a critical vulnerability that would not pass my security check.

[jzzzzzzzzz-z]

Sharing a working solution in EKS deployment:

Attach the application service account with a AWS IAM role that has access your AWS OpenSearch instance, config your helm values like the following

openmetadata:
  config:
    elasticsearch:
      enabled: true
      host: localhost
      searchType: opensearch
      port: 8080
      scheme: http
      connectionTimeoutSecs: 5
      socketTimeoutSecs: 60
      keepAliveTimeoutSecs: 600
      batchSize: 10
      auth:
        enabled: false


sidecars:
- name: "aws-sigv4-proxy"
  image: "<aws-sigv4-proxy-image> https://github.com/awslabs/aws-sigv4-proxy"
  imagePullPolicy: "Always"
  command: ["./aws-sigv4-proxy"]
  args: 
  - "--verbose"
  - "--log-failed-requests"
  - "--log-signing-process"
  - "--no-verify-ssl"
  - "--name"
  - "es"
  - "--region"
  - "<Your AWS Region>"
  - "--host"
  - "<Your AWS Opensearch Host>"
  - "--sign-host"
  - "<Your AWS Opensearch Host>"
  env:
  - name: AWS_SDK_LOAD_CONFIG
    value: "true"

[luqmansen]

Seems like this is going to be supported very soon without an extra sidecar proxy

• Fix AWS OpenSearch IAM authentication with AwsSdk2Transport #25768