adot EKS Addon Version v0.109.0-eksbuild.2 causes repeated log line Error (2x/second) when http protocol otlp receiver is used

[colinbjohnson]

Describe the bug

Version v0.109.0-eksbuild.2 of the adot EKS addon utilizes the image public.ecr.aws/aws-observability/aws-otel-collector:v0.41.1. Under certain conditions (believed to be if a receiver is configured as follows:

receivers:
  otlp:
    protocols:
      http:

The following error is thrown repeatedly: http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/internal/request.(*RespWriterWrapper).writeHeader (resp_writer_wrapper.go:78).

In order to resolve this issue it appears we would need to update to public.ecr.aws/aws-observability/aws-otel-collector:v0.42.0 however it appears that there is no way to do this.

Steps to reproduce

See #2881.

What did you expect to see?

I would expect to see either:

1. An ability to modify the image version to (public.ecr.aws/aws-observability/aws-otel-collector:v0.42.0) when running the addon
2. A release of the adot addon without this issue

What did you see instead?

There is no way to fix this issue at present.

Environment
Describe any aspect of your environment relevant to the problem.
If this is related to a deployment of the ADOT Collector please
provide your Collector config file.

Additional context
Add any other context about the problem here.

[colinbjohnson]

It turns out that version v0.109.0-eksbuild.2 is not compatible with EKS 1.31. For customers running this configuration these are the choices:

1. Move off of the EKS Add-on.
2. Pay extra EKS costs for extended support.
3. Disable whichever conditions (believed to be "enabling the otlp HTTP receiver") cause the repeated log lines and accept the loss of functionality.
4. Accept the log lines being generated repeatedly and pay for the extra costs.