[exporter/elasticsearch] - Document level 409 Conflict responses are unconditionally logged as errors

[cmacknz]

Component(s)

exporter/elasticsearch

What happened?

Description

Document level 409 Conflict responses are unconditionally logged as errors. Use of document IDs as form of de-duplication where consecutive writes to a document that already exist fail with an expected and ignorable 409 Conflict or "version conflict exception" is very common. As the collector has been incorporated into more of Elastic's products, users have starting seeing these error logs in non-error cases and filing bugs or support cases about it.

This pattern is common enough that the elasticsearch exporter already has special handling for this case for the profiling signal:

opentelemetry-collector-contrib/exporter/elasticsearchexporter/bulkindexer.go

Lines 393 to 400 in 12a1810

	if resp.Error.Type == "version_conflict_engine_exception" &&
	(strings.HasPrefix(resp.Index, ".profiling-stackframes-") ||
	strings.HasPrefix(resp.Index, ".profiling-stacktraces-")) {
	// For the Profiling indices .profiling-[stacktraces|stackframes]- the
	// rejection of duplicates are expected from Elasticsearch. So we do not want
	// to log these here.
	continue
	}

Other clients, like Beats, simply record the number of duplicates with a metric and do not log it as can be seen here. This has at times covered up unexpected 409 responses as a trade off, such as incorrectly configured TSDB index dimensions. At the same time, "at least once" retry strategies will generate ignorable 409 responses.

There needs to be a way to opt out (or into) logging of document level conflict responses to handle the very common case where a 409 conflict is not an error.

An initial suggestion would be to control whether 409 conflicts are logged at all with a dedicated configuration flag that can be toggled independent of the log level, so that cases that produce many duplicate responses as part of normal operation do not have their debug logs flooded with uninteresting document level logging. If additional configuration is not wanted, logging nothing is better than logging unnecessarily considering the high rate of logs that could be generated. Using a rate limited debug logger would be an acceptable solution as well.

If there is no already, 409 Conflict responses should have a dedicated metric (it looks like we get this via the existing attribute.String("error.type", resp.Error.Type) on the documents processed metric).

Collector version

0.145.0

Environment information

All

OpenTelemetry Collector configuration

Log output

{"log.level":"error","@timestamp":"2026-02-25T14:43:22.497+0100","message":"failed to index document","log.origin":{"file.line":391,"file.name":"elasticsearchexporter@v0.141.0/bulkindexer.go","function":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer"},"otelcol.signal":"logs","error.type":"version_conflict_engine_exception","error.reason":"[605lwQVhSGaYEsZAQkzefQ:Ry_5ihrzScikgrsFS5WgOg:.ds-logs-endpoint.events.file-default-2025.11.28-000025:s0:p]: version conflict, document already exists (current version [1])","log.origin.stack_trace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.141.0/bulkindexer.go:391\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*syncBulkIndexerSession).Flush\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.141.0/bulkindexer.go:216\ngithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*sessionList).Flush.func1\n\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.141.0/exporter.go:613\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.18.0/errgroup/errgroup.go:93","resource":{"service.instance.id":"5be2fd74-7efc-4f7e-be26-ddaca91fa4be","service.name":"/usr/share/elastic-agent/data/elastic-agent-356494/components/elastic-otel-collector","service.version":"9.3.0"},"otelcol.component.id":"elasticsearch/_agent-component/default","otelcol.component.kind":"exporter","index":".ds-metrics-elasticsearch.stack_monitoring.shard-default-2026.02.25-000001","hint":"check the \"Known issues\" section of Elasticsearch Exporter docs","ecs.version":"1.6.0"}

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

/label -needs-triage