Issue with ElasticSearch - Cadence Server Frontend 1.3.1

[CosminL-DEV]

ElasticSearch Configuration Error with Cadence Web (cadence-server-frontend) - Unable to View Workflows

Issue Description

I have installed ElasticSearch on my Kubernetes cluster for both v7 and v8 (always configuring v7 in the config_template since v8 is not recognized). When trying to view workflows from cadence-web (it's a cadence-frontend error when it's doing the query), I'm getting an error.

Configuration

config_template.yaml

# Log configuration
log:
  stdout: true
  level: info
  levelKey: level
  encoding: json

# Persistence configuration
persistence:
  numHistoryShards: 4
  defaultStore: "default"
  advancedVisibilityStore: "es-visibility"
  enablePersistenceLatencyHistogramMetrics: false

  # DataStores configuration
  datastores:
    # Default datastore
    default:
      nosql:
        pluginName: "cassandra"
        hosts: "cassandra.cadence-testing.svc.cluster.local"
        keyspace: "cadence"
        user: "cassandra"
        password: {{ .Env.CASSANDRA_PASSWORD }}
        protoVersion: 4
        maxConns: 10
        connectTimeout: "10s"
        timeout: "1s"
        consistency: "LOCAL_QUORUM"
        serialConsistency: "LOCAL_SERIAL"
        hostSelectionPolicy: "tokenaware,roundrobin"
      
    # Elasticsearch for advanced visibility
    es-visibility:
      elasticsearch:
        disableSniff: true
        version: "v7"
        url:
          scheme: "https"
          host: elasticsearch.cadence-testing.svc.cluster.local:9200
        indices:
          visibility: "cadence-visibility"
        username: "elastic"
        password: {{ .Env.ES_PWD }}
        tls:
          enabled: true
          caFile: "/etc/cadence/ssl/ca.pem"
          certFile: "/etc/cadence/ssl/client.pem"
          keyFile: "/etc/cadence/ssl/client-key.pem"
          enableHostVerification: true
          requireClientAuth: false

# Ringpop configuration
ringpop:
  name: cadence
  bootstrapMode: dns
  bootstrapHosts:
    - cadence-frontend-headless:7933
    - cadence-history-headless:7934
    - cadence-matching-headless:7935
    - cadence-worker-headless:7939
  maxJoinDuration: 30s

# Cluster configuration
clusterGroupMetadata:
  failoverVersionIncrement: 10
  primaryClusterName: cluster0
  currentClusterName: cluster0
  clusterGroup:
    cluster0:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: cadence-frontend:7833
      rpcTransport: "grpc"
  clusterRedirectionPolicy:
    policy: noop

# Services configuration
services:
  frontend:
    rpc:
      port: 7933
      grpcPort: 7833
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  history:
    rpc:
      port: 7934
      grpcPort: 7834
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  matching:
    rpc:
      port: 7935
      grpcPort: 7835
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  worker:
    rpc:
      port: 7939
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

# Kafka configuration
kafka:
  tls:
    enabled: false
    enableHostVerification: true
    requireClientAuth: false
  sasl:
    enabled: false
  clusters:
    default:
      brokers:
        - kafka-cluster-kafka-bootstrap.strimzi.svc.cluster.local:9092
  topics:
    cadence-visibility:
      cluster: default
    cadence-visibility-dlq:
      cluster: default
  applications:
    visibility:
      topic: cadence-visibility
      dlq-topic: cadence-visibility-dlq


# Archival configuration
archival:
  history:
    status: "disabled"
    enableRead: false
    provider:
      filestore:
        fileMode: "0644"
        dirMode: "0755"
  visibility:
    status: "disabled"
    enableRead: false
    provider:
      filestore:
        fileMode: "0644"
        dirMode: "0755"

# Domain defaults configuration
domainDefaults:
  archival:
    history:
      status: "disabled"
    visibility:
      status: "disabled"

# Blobstore configuration
blobstore:
  filestore:
    outputDirectory: "/etc/cadence/blobstore"

# Public client configuration
publicClient:
  transport: "grpc"
  RefreshInterval: "10s"

# Dynamic configuration
dynamicconfig:
  client: "filebased"
  filebased:
    filepath: "/etc/cadence/config/dynamicconfig/config.yaml"
    pollInterval: "60s"

Dynamic Configs

frontend.enableClientVersionCheck:
- value: true
history.EnableConsistentQueryByDomain:
- constraints: {}
  value: true
system.minRetentionDays:
- constraints: {}
  value: 0
system.readVisibilityStoreName:
- value: es
system.writeVisibilityStoreName:
- value: es

Environment Variables

I only have the following environment variables configured:

• SERVICES
• Cassandra password
• ElasticSearch password

Error Log (Cadence-Frontend)

{"level":"error","ts":"2025-06-24T13:21:00.569Z","msg":"Deprecation warning: 299 Elasticsearch-8.18.0-04e979aa50b657bebd4a0937389308de82c2bdad \"Deprecated field [to] used, this field is unused and will be removed entirely\"","service":"cadence-frontend","component":"visibility-client","logging-call-at":"client.go:50","stacktrace":"github.com/uber/cadence/common/elasticsearch/client/v7.convertlogger.Printf\n\t/cadence/common/elasticsearch/client/v7/client.go:50\ngithub.com/olivere/elastic/v7.(*Client).errorf\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.21/client.go:822\ngithub.com/olivere/elastic/v7.(*Client).PerformRequest\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.21/client.go:1416\ngithub.com/olivere/elastic/v7.(*SearchService).Do\n\t/go/pkg/mod/github.com/olivere/elastic/v7@v7.0.21/search.go:620\ngithub.com/uber/cadence/common/elasticsearch/client/v7.(*ElasticV7).Search\n\t/cadence/common/elasticsearch/client/v7/client.go:114\ngithub.com/uber/cadence/common/elasticsearch.(*ESClient).SearchByQuery\n\t/cadence/common/elasticsearch/client.go:90\ngithub.com/uber/cadence/common/persistence/elasticsearch.(*esVisibilityStore).ListWorkflowExecutions\n\t/cadence/common/persistence/elasticsearch/es_visibility_store.go:426\ngithub.com/uber/cadence/common/persistence.(*visibilityManagerImpl).ListWorkflowExecutions\n\t/cadence/common/persistence/visibility_single_manager.go:300\ngithub.com/uber/cadence/common/persistence/wrappers/ratelimited.(*ratelimitedVisibilityManager).ListWorkflowExecutions\n\t/cadence/common/persistence/wrappers/ratelimited/visibility_generated.go:133\ngithub.com/uber/cadence/common/persistence/elasticsearch.(*visibilityMetricsClient).ListWorkflowExecutions\n\t/cadence/common/persistence/elasticsearch/es_visibility_metric_clients.go:288\ngithub.com/uber/cadence/common/persistence.(*visibilityHybridManager).ListWorkflowExecutions\n\t/cadence/common/persistence/visibility_hybrid_manager.go:498\ngithub.com/uber/cadence/service/frontend/api.(*WorkflowHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/api/list_workflow_handlers.go:358\ngithub.com/uber/cadence/service/frontend/wrappers/versioncheck.(*versionCheckHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/versioncheck/api_generated.go:161\ngithub.com/uber/cadence/service/frontend/wrappers/ratelimited.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/ratelimited/api_generated.go:250\ngithub.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/metered/api_generated.go:305\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions.func2\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:389\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*noopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/wrappers/clusterredirection/policy.go:189\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:385\ngithub.com/uber/cadence/service/frontend/wrappers/accesscontrolled.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/accesscontrolled/api_generated.go:304\ngithub.com/uber/cadence/service/frontend/wrappers/grpc.APIHandler.ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/grpc/api_generated.go:105\ngithub.com/uber/cadence-idl/go/proto/api/v1.(*_VisibilityAPIYARPCHandler).ListWorkflowExecutions\n\t/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20250521225400-08210635c303/go/proto/api/v1/service_visibility.pb.yarpc.go:351\ngo.uber.org/yarpc/encoding/protobuf.(*unaryHandler).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/encoding/protobuf/inbound.go:56\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:72\ngithub.com/uber/cadence/common/rpc.(*ForwardPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:224\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*ClientPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:261\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*InboundMetricsMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:116\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*PinotComparatorMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:132\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/observability/middleware.go:188\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChain.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:60\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/grpc.(*handler).callUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:240\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnaryBeforeErrorConversion\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:232\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:190\ngo.uber.org/yarpc/transport/grpc.(*handler).handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:77\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1637\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1747\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"}
{"level":"error","ts":"2025-06-24T13:21:00.569Z","msg":"Operation failed with internal error.","service":"cadence-frontend","metric-scope":309,"error":"ListWorkflowExecutions failed, elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","logging-call-at":"es_visibility_metric_clients.go:382","stacktrace":"github.com/uber/cadence/common/persistence/elasticsearch.(*visibilityMetricsClient).updateErrorMetric\n\t/cadence/common/persistence/elasticsearch/es_visibility_metric_clients.go:382\ngithub.com/uber/cadence/common/persistence/elasticsearch.(*visibilityMetricsClient).ListWorkflowExecutions\n\t/cadence/common/persistence/elasticsearch/es_visibility_metric_clients.go:292\ngithub.com/uber/cadence/common/persistence.(*visibilityHybridManager).ListWorkflowExecutions\n\t/cadence/common/persistence/visibility_hybrid_manager.go:498\ngithub.com/uber/cadence/service/frontend/api.(*WorkflowHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/api/list_workflow_handlers.go:358\ngithub.com/uber/cadence/service/frontend/wrappers/versioncheck.(*versionCheckHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/versioncheck/api_generated.go:161\ngithub.com/uber/cadence/service/frontend/wrappers/ratelimited.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/ratelimited/api_generated.go:250\ngithub.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/metered/api_generated.go:305\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions.func2\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:389\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*noopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/wrappers/clusterredirection/policy.go:189\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:385\ngithub.com/uber/cadence/service/frontend/wrappers/accesscontrolled.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/accesscontrolled/api_generated.go:304\ngithub.com/uber/cadence/service/frontend/wrappers/grpc.APIHandler.ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/grpc/api_generated.go:105\ngithub.com/uber/cadence-idl/go/proto/api/v1.(*_VisibilityAPIYARPCHandler).ListWorkflowExecutions\n\t/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20250521225400-08210635c303/go/proto/api/v1/service_visibility.pb.yarpc.go:351\ngo.uber.org/yarpc/encoding/protobuf.(*unaryHandler).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/encoding/protobuf/inbound.go:56\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:72\ngithub.com/uber/cadence/common/rpc.(*ForwardPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:224\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*ClientPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:261\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*InboundMetricsMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:116\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*PinotComparatorMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:132\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/observability/middleware.go:188\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChain.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:60\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/grpc.(*handler).callUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:240\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnaryBeforeErrorConversion\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:232\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:190\ngo.uber.org/yarpc/transport/grpc.(*handler).handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:77\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1637\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1747\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"}
{"level":"error","ts":"2025-06-24T13:21:00.569Z","msg":"Internal service error","service":"cadence-frontend","wf-handler-name":"ListWorkflowExecutions","wf-domain-name":"default","error":"ListWorkflowExecutions failed, elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","logging-call-at":"metered.go:51","stacktrace":"github.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).handleErr\n\t/cadence/service/frontend/wrappers/metered/metered.go:51\ngithub.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/metered/api_generated.go:307\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions.func2\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:389\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*noopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/wrappers/clusterredirection/policy.go:189\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:385\ngithub.com/uber/cadence/service/frontend/wrappers/accesscontrolled.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/accesscontrolled/api_generated.go:304\ngithub.com/uber/cadence/service/frontend/wrappers/grpc.APIHandler.ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/grpc/api_generated.go:105\ngithub.com/uber/cadence-idl/go/proto/api/v1.(*_VisibilityAPIYARPCHandler).ListWorkflowExecutions\n\t/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20250521225400-08210635c303/go/proto/api/v1/service_visibility.pb.yarpc.go:351\ngo.uber.org/yarpc/encoding/protobuf.(*unaryHandler).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/encoding/protobuf/inbound.go:56\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:72\ngithub.com/uber/cadence/common/rpc.(*ForwardPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:224\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*ClientPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:261\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*InboundMetricsMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:116\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*PinotComparatorMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:132\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/observability/middleware.go:188\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChain.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:60\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/grpc.(*handler).callUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:240\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnaryBeforeErrorConversion\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:232\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:190\ngo.uber.org/yarpc/transport/grpc.(*handler).handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:77\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1637\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1747\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"}

Expected Behavior

The workflows should be visible in cadence-web without errors.

Actual Behavior

Error occurs when attempting to view workflows through the web interface (it connects to cadence-frontend).

Environment Details

• Kubernetes cluster: 1.32
• ElasticSearch version: v7 and v8 tested. With and without TLS
• Cadence version: 1.3.1

[CosminL-DEV]

@taylanisikdemir Can you check this with the TEAM if cadence 1.3.1 is still working with ElasticSearch? I wan't to check if it's a Kubernetes/Helm Chart problem or it's a cadence server/ Elastic Schema problem. Thank you!

[neil-xie]

Hi, there is no change related to ES in v1.3.1, I will try running this and get back to you tomorrow.

[CosminL-DEV]

Hi @taylanisikdemir .
@neil-xie I'm sharing my full step-step

Debug Pod Configuration

apiVersion: v1
kind: Pod
metadata:
  name: cadence-debug-pod
  namespace: cadence-testing
  labels:
    app: cadence-debug
spec:
  restartPolicy: Always
  containers:
  - name: cadence-server
    image: docker.io/ubercadence/server:v1.3.2
    imagePullPolicy: IfNotPresent
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 3600; done"]
    env:
      - name: KEYSPACE
        value: cadence
      - name: VISIBILITY_KEYSPACE
        value: cadence_visibility
      - name: RF
        value: "1"
      - name: CADENCE_HOME
        value: /etc/cadence

ElasticSearch Installation

git clone git@github.com:elastic/helm-charts.git
cd helm-charts
git checkout 7.9
# Note: Changed API version in ./templates/poddisruptionbudget.yaml from policy/v1beta1 to policy/v1
helm install elasticsearch ./elasticsearch --set imageTag=7.9.3 -n cadence-testing

Schema Installation

# ElasticSearch Schema Setup using debug pod
SCHEMA_FILE=$CADENCE_HOME/schema/elasticsearch/v7/visibility/index_template.json
server="elasticsearch-master.cadence-testing.svc.cluster.local:9200"
URL="http://$server/_template/cadence-visibility-template"
curl -X PUT $URL -H 'Content-Type: application/json' --data-binary "@$SCHEMA_FILE"
URL="http://$server/cadence-visibility"
curl -X PUT $URL

Values.yaml

Deployment Status

• ElasticSearch: ✅ Successfully deployed
• Cadence: ✅ Successfully deployed with helm install cadence -n cadence-testing .
• Schema: ✅ Applied successfully

Deployed cadence-config

# Log configuration
log:
  stdout: true
  level: info
  levelKey: level
  encoding: json

# Persistence configuration
persistence:
  numHistoryShards: 4
  defaultStore: "default"
  advancedVisibilityStore: "es-visibility"
  enablePersistenceLatencyHistogramMetrics: false

  # DataStores configuration
  datastores:
    # Default datastore
    default:
      nosql:
        pluginName: "cassandra"
        hosts: "cassandra-service.cadence-testing.svc.cluster.local"
        keyspace: "cadence"
        user: "cassandra"
        password: {{ .Env.CASSANDRA_PASSWORD }}
        protoVersion: 4
        maxConns: 10
        connectTimeout: "10s"
        timeout: "1s"
        consistency: "LOCAL_QUORUM"
        serialConsistency: "LOCAL_SERIAL"
        hostSelectionPolicy: "tokenaware,roundrobin"
      
    # Elasticsearch for advanced visibility
    es-visibility:
      elasticsearch:
        disableSniff: true
        version: "v7"
        url:
          scheme: "http"
          host: elasticsearch-master.cadence-testing.svc.cluster.local:9200
        indices:
          visibility: "cadence-visibility"

# Ringpop configuration
ringpop:
  name: cadence
  bootstrapMode: dns
  bootstrapHosts:
    - cadence-frontend-headless:7933
    - cadence-history-headless:7934
    - cadence-matching-headless:7935
    - cadence-worker-headless:7939
  maxJoinDuration: 30s

# Cluster configuration
clusterGroupMetadata:
  failoverVersionIncrement: 10
  primaryClusterName: cluster0
  currentClusterName: cluster0
  clusterGroup:
    cluster0:
      enabled: true
      initialFailoverVersion: 0
      rpcAddress: cadence-frontend:7833
      rpcTransport: "grpc"
  clusterRedirectionPolicy:
    policy: noop

# Services configuration
services:
  frontend:
    rpc:
      port: 7933
      grpcPort: 7833
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  history:
    rpc:
      port: 7934
      grpcPort: 7834
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  matching:
    rpc:
      port: 7935
      grpcPort: 7835
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

  worker:
    rpc:
      port: 7939
      bindOnIP: {{ default .Env.POD_IP "0.0.0.0" }}
      grpcMaxMsgSize: 4194304
    metrics:
      prometheus:
        timerType: histogram
        listenAddress: "0.0.0.0:9090"

# Kafka configuration
kafka:
  tls:
    enabled: false
    enableHostVerification: true
    requireClientAuth: false
  sasl:
    enabled: false
  clusters:
    default:
      brokers:
        - kafka-cluster-kafka-bootstrap.strimzi.svc.cluster.local:9092
  topics:
    cadence-visibility:
      cluster: default
    cadence-visibility-dlq:
      cluster: default
  applications:
    visibility:
      topic: cadence-visibility
      dlq-topic: cadence-visibility-dlq


# Archival configuration
archival:
  history:
    status: "disabled"
    enableRead: false
    provider:
      filestore:
        fileMode: "0644"
        dirMode: "0755"
  visibility:
    status: "disabled"
    enableRead: false
    provider:
      filestore:
        fileMode: "0644"
        dirMode: "0755"

# Domain defaults configuration
domainDefaults:
  archival:
    history:
      status: "disabled"
    visibility:
      status: "disabled"

# Blobstore configuration
blobstore:
  filestore:
    outputDirectory: "/etc/cadence/blobstore"

# Public client configuration
publicClient:
  transport: "grpc"
  RefreshInterval: "10s"

# Dynamic configuration
dynamicconfig:
  client: "filebased"
  filebased:
    filepath: "/etc/cadence/config/dynamicconfig/config.yaml"
    pollInterval: "60s"

Error Details

Error Message

Full Web Log

{
  "level": "info",
  "time": 1752567999429,
  "pid": 1,
  "hostname": "cadence-web-6557dd7f96-5wdrv",
  "name": "console",
  "message": "✓ Ready in 253ms"
}
{
  "level": "error",
  "time": 1752568215270,
  "pid": 1,
  "hostname": "cadence-web-6557dd7f96-5wdrv",
  "requestParams": {
    "domain": "default",
    "cluster": "cluster0"
  },
  "queryParams": {
    "pageSize": 20,
    "listType": "default",
    "inputType": "search",
    "search": "",
    "timeColumn": "StartTime",
    "timeRangeStart": "2025-06-15T08:30:29.908Z",
    "timeRangeEnd": "2025-07-15T08:30:29.908Z",
    "sortColumn": "StartTime",
    "sortOrder": "DESC"
  },
  "error": {
    "type": "i",
    "message": "cadence internal error, msg: ListWorkflowExecutions failed, elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]",
    "grpcStatusCode": 2,
    "name": "GRPCError"
  }
}

Full Frontend log:

{"level":"error","ts":"2025-07-15T08:30:26.563Z","msg":"Operation failed with internal error.","service":"cadence-frontend","metric-scope":310,"error":"ListWorkflowExecutions failed, elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","logging-call-at":"es_visibility_metric_clients.go:382","stacktrace":"github.com/uber/cadence/common/persistence/elasticsearch.(*visibilityMetricsClient).updateErrorMetric\n\t/cadence/common/persistence/elasticsearch/es_visibility_metric_clients.go:382\ngithub.com/uber/cadence/common/persistence/elasticsearch.(*visibilityMetricsClient).ListWorkflowExecutions\n\t/cadence/common/persistence/elasticsearch/es_visibility_metric_clients.go:292\ngithub.com/uber/cadence/common/persistence.(*visibilityHybridManager).ListWorkflowExecutions\n\t/cadence/common/persistence/visibility_hybrid_manager.go:498\ngithub.com/uber/cadence/service/frontend/api.(*WorkflowHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/api/list_workflow_handlers.go:358\ngithub.com/uber/cadence/service/frontend/wrappers/versioncheck.(*versionCheckHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/versioncheck/api_generated.go:161\ngithub.com/uber/cadence/service/frontend/wrappers/ratelimited.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/ratelimited/api_generated.go:250\ngithub.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/metered/api_generated.go:305\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions.func2\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:389\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*noopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/wrappers/clusterredirection/policy.go:189\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:385\ngithub.com/uber/cadence/service/frontend/wrappers/accesscontrolled.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/accesscontrolled/api_generated.go:304\ngithub.com/uber/cadence/service/frontend/wrappers/grpc.APIHandler.ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/grpc/api_generated.go:105\ngithub.com/uber/cadence-idl/go/proto/api/v1.(*_VisibilityAPIYARPCHandler).ListWorkflowExecutions\n\t/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20250616185004-cc6f52f87bc6/go/proto/api/v1/service_visibility.pb.yarpc.go:351\ngo.uber.org/yarpc/encoding/protobuf.(*unaryHandler).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/encoding/protobuf/inbound.go:56\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:72\ngithub.com/uber/cadence/common/rpc.(*ForwardPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:224\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*ClientPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:261\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*InboundMetricsMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:116\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*PinotComparatorMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:132\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/observability/middleware.go:188\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChain.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:60\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/grpc.(*handler).callUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:240\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnaryBeforeErrorConversion\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:232\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:190\ngo.uber.org/yarpc/transport/grpc.(*handler).handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:77\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1637\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1747\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"}
{"level":"error","ts":"2025-07-15T08:30:26.563Z","msg":"Internal service error","service":"cadence-frontend","wf-handler-name":"ListWorkflowExecutions","wf-domain-name":"default","error":"ListWorkflowExecutions failed, elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]","logging-call-at":"metered.go:51","stacktrace":"github.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).handleErr\n\t/cadence/service/frontend/wrappers/metered/metered.go:51\ngithub.com/uber/cadence/service/frontend/wrappers/metered.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/metered/api_generated.go:307\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions.func2\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:389\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*noopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/wrappers/clusterredirection/policy.go:189\ngithub.com/uber/cadence/service/frontend/wrappers/clusterredirection.(*clusterRedirectionHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/clusterredirection/api_generated.go:385\ngithub.com/uber/cadence/service/frontend/wrappers/accesscontrolled.(*apiHandler).ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/accesscontrolled/api_generated.go:304\ngithub.com/uber/cadence/service/frontend/wrappers/grpc.APIHandler.ListWorkflowExecutions\n\t/cadence/service/frontend/wrappers/grpc/api_generated.go:105\ngithub.com/uber/cadence-idl/go/proto/api/v1.(*_VisibilityAPIYARPCHandler).ListWorkflowExecutions\n\t/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20250616185004-cc6f52f87bc6/go/proto/api/v1/service_visibility.pb.yarpc.go:351\ngo.uber.org/yarpc/encoding/protobuf.(*unaryHandler).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/encoding/protobuf/inbound.go:56\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:72\ngithub.com/uber/cadence/common/rpc.(*ForwardPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:224\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*ClientPartitionConfigMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:261\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*InboundMetricsMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:116\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngithub.com/uber/cadence/common/rpc.(*PinotComparatorMiddleware).Handle\n\t/cadence/common/rpc/middleware.go:132\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/observability/middleware.go:188\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChainExec.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:76\ngo.uber.org/yarpc/internal/inboundmiddleware.unaryChain.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/internal/inboundmiddleware/chain.go:60\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/grpc.(*handler).callUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:240\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnaryBeforeErrorConversion\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:232\ngo.uber.org/yarpc/transport/grpc.(*handler).handleUnary\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:190\ngo.uber.org/yarpc/transport/grpc.(*handler).handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.70.3/transport/grpc/handler.go:77\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1637\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1747\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986"}

I go to port forward elasticsearch and the path /cadence-visibility

[CosminL-DEV]

@taylanisikdemir Temporal had the same issue and I think it's unsolved :(
https://community.temporal.io/t/elasticsearch-listworkflowexecutions-failed-no-mapping-found-for-closetime/9483/6

Have a look, maybe I should just comment that section and say it's currently not supported in helm chart version of cadence. But I would like to let there for the future because it's already done the configuration with elastic and cadence it's reading correctly, it's seems more like an schema or a communication error cadence-elastic

[neil-xie]

Were you index requests working correctly? I have got the same error locally caused by mapping issue, I had to delete the index and recreate index.