Add Elasticsearch exporter

[dmitry-naviteq]

Is your feature request related to a problem? Please describe.
the upstream otel has exporter to elasticsearch that we use in current setup, but in ADOT it does not exist

Describe the solution you'd like
expected anything in upstream to exist as optional features that we can keep using when moving from upstream collectors to ADOT and EKS

Describe alternatives you've considered
just wondering why some of the original exporters and features are being removed, isn't it easier to just leave them as-is when developing the 'additional' ADOT features, do they conflict with each other in some way?

Additional context
here's the original:
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter

[mhausenblas]

Thanks for raising this feature request. To be clear: nothing was "removed". We select certain components from upstream contrib (see https://github.com/aws-observability/aws-otel-collector?tab=readme-ov-file#adot-collector-built-in-components for the detailed list that is included in the ADOT collector) and the ES exporter never was part of our collector. I hence interpret your issue here as a feature request to add it.

[dmitry-naviteq]

Thanks for raising this feature request. To be clear: nothing was "removed". We select certain components from upstream contrib (see https://github.com/aws-observability/aws-otel-collector?tab=readme-ov-file#adot-collector-built-in-components for the detailed list that is included in the ADOT collector) and the ES exporter never was part of our collector. I hence interpret your issue here as a feature request to add it.

thanks, I thought the idea of ADOT was to have all the features of the upstream and add more to it? otherwise people would need to use multiple installations of the otel collectors, one ADOT for the xray and custom aws features, and one upstream for everything else? it seems not optimal. I wonder what was the benefit of making ADOT limited in features, does it consume more resources if all upstream features will be included as is?

[dmitry-naviteq]

I noticed the k8s receivers of kubeletstats and k8s_cluster are also not included, just going through the official docs https://opentelemetry.io/docs/kubernetes/collector/components/#kubeletstats-receiver
and wondering why ADOT is a default addon for EKS, but k8s receivers like these aren't bundled by default, isn't it useful to have cluster metadata when running in EKS?

what can I use instead, to populate all traces with k8s metadata? I'm probably missing something, but haven't found much in the official docs here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-OpenTelemetry.html
it only has prometheus example but nothing about how to include k8s metadata for traces and metrics 🤔

appreciate your help 😊

[mhausenblas]

We select components based on use cases supported, driven by customer demand. The key values are: 1. it's supported (if you have Enterprise support), and 2. security (think patching CVEs etc.).

If you want full flexibility and pick and choose which component, I recommend that you use OCB.

[dmitry-naviteq]

We select components based on use cases supported, driven by customer demand. The key values are: 1. it's supported (if you have Enterprise support), and 2. security (think patching CVEs etc.).

If you want full flexibility and pick and choose which component, I recommend that you use OCB.

thanks! my goal is to have traces through "ALB -> EKS -> pods / namespaces / nodes (names included, for x-ray let's say, so we can find in which pod what originates) -> SQS / SNS / Kinesis / S3 -> back to client", which collector is best to use in this case? If I remove ADOT I lose the option to stream this to x-ray etc'? was hoping ADOT is best option to inspect all these aws services in traces. (my plan was to stream let's say 10% sample rate to elasticsearch, and 5% sample rate to x-ray, just to catch any aws related permission errors and so on).

[mhausenblas]

which collector is best to use in this case?

Sounds like you are best of with DIY collector based on OCB.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.