Skip to content

[Bug]: elasticsearch.env helper always injects default credentials, breaking AWS OpenSearch #742

@domolitom

Description

@domolitom

What happened?

The elasticsearch.env helper in _helpers.tpl unconditionally injects ES_USERNAME (default: elastic) and ES_PASSWORD (default: changeme) into the ES maintenance jobs (esIndexCleaner, esRollover, esLookback).

This breaks AWS OpenSearch deployments that rely on VPC-level access without basic auth. AWS rejects the malformed Authorization header with a 403.

This is a regression — the storage.elasticsearch.anonymous option was added in PR #442 (chart v0.67.1) to address exactly this scenario (issue #441), but it was lost when the chart was rewritten for Jaeger v2.

There is no way to prevent these env vars from being set via values alone:

  • Setting user: "" doesn't work because Go template default treats empty string as falsy
  • Using extraEnv to override with duplicate names fails with Kubernetes strategic merge patch errors

Steps to reproduce

  1. Deploy Jaeger chart (v4.5.0) with esIndexCleaner.enabled: true pointing at AWS OpenSearch using VPC-level access (no basic auth):
storage:
  type: elasticsearch
  elasticsearch:
    url: "https://vpc-my-domain.eu-west-1.es.amazonaws.com:443"

esIndexCleaner:
  enabled: true
  numberOfDays: 7
  1. Wait for the CronJob to run (or trigger it manually)
  2. The job fails with 403

Expected behavior

A way to disable credential injection for ES maintenance jobs, like the storage.elasticsearch.anonymous: true option that existed before the v2 rewrite (PR #442).

Relevant log output

failed to query indices: request failed, status code: 403, body: {"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header."}

Additional context

Current code in _helpers.tpl always injects credentials with hardcoded defaults:

{{- $user := $es.user | default "elastic" -}}
{{- $password := $es.password | default "changeme" -}}
- name: ES_USERNAME
  value: {{ $user | quote }}
- name: ES_PASSWORD
  value: {{ $password | quote }}

Proposed fix in PR #743: re-add {{- if not $es.anonymous }} guard around the credential env vars, matching the approach from PR #442.

Jaeger setup

Jaeger backend version: 2.14.1

Storage backend: AWS OpenSearch (VPC endpoint, no fine-grained access control)

Operating system: Linux (EKS)

Deployment model: Kubernetes via Helm chart v4.4.2

Deployment configs:

# values.yaml
storage:
  type: elasticsearch
  elasticsearch:
    url: "https://vpc-my-domain.eu-west-1.es.amazonaws.com:443"

esIndexCleaner:
  enabled: true
  numberOfDays: 7

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions