fix(kafkareceiver): enforce a backoff mechanism on exporterhelper.ErrQueueIsFull error #39581
Conversation
| @@ -851,7 +852,8 @@ func newExponentialBackOff(config configretry.BackOffConfig) *backoff.Exponentia | |||
| } | |||
|
|
|||
| func errorRequiresBackoff(err error) bool { | |||
| return err.Error() == errMemoryLimiterDataRefused.Error() | |||
| return err.Error() == errMemoryLimiterDataRefused.Error() || | |||
There was a problem hiding this comment.
Can this used either errors.Is or errors.As instead of comparing the strings?
There was a problem hiding this comment.
Unfortunately, I don't see how this can be utilized without changes in opentelemetry-collector, since the memory limiter error is in an internal package: https://github.com/open-telemetry/opentelemetry-collector/blob/main/internal/memorylimiter/memorylimiter.go#L28
|
Moving to draft while this is being worked on |
There was a problem hiding this comment.
@an-mmx did you consider updating the logic to retry on any error that is not considered permanent, like the exporterhelper retry sender? https://github.com/open-telemetry/opentelemetry-collector/blob/d020c9074f873c54e8ae7b8eaa4a08e13157cb76/exporter/exporterhelper/internal/retry_sender.go#L96-L99
I think that would be the ideal solution
I hadn’t considered this approach before. |
|
@MovieStoreGuy @pavolloffay any concerns with the suggestion above to retry on any non-permanent errors? I expect we'll need to go through and classify some errors as permanent, but I don't think that needs to block? |
an-mmx
force-pushed
the
fix/kafkareceiver-backoff-on-queue-batch
branch
from
b18be1c to
e5dcad8
Compare
|
Changed to enforce a backoff retry on any non permanent error |
an-mmx
force-pushed
the
fix/kafkareceiver-backoff-on-queue-batch
branch
4 times, most recently
from
b8a43f0 to
8b699d3
Compare
an-mmx
force-pushed
the
fix/kafkareceiver-backoff-on-queue-batch
branch
from
8b699d3 to
36bf22d
Compare
an-mmx
force-pushed
the
fix/kafkareceiver-backoff-on-queue-batch
branch
from
774127e to
f5bb920
Compare
|
Would you please rebase your branch and update the collector module versions? See the failing check, you can apply the git diff. |
an-mmx
force-pushed
the
fix/kafkareceiver-backoff-on-queue-batch
branch
from
f5bb920 to
933dec4
Compare
…QueueIsFull error (open-telemetry#39581) #### Description In the current implementation, the BackOff mechanism is trigerred only by [memory limiter error](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kafkareceiver/kafka_receiver.go#L562-L564). However, the same behavior is expected for the `exporterhelper.ErrQueueIsFull` error, which occurs when there is a sending queue overflow (both [memory queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queuebatch/memory_queue.go#L109-L112) and [persistent queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/internal/queuebatch/persistent_queue.go#L244-L247)). #### Link to tracking issue open-telemetry#39580 #### Testing Unit coverage added #### Documentation No documentation updated --------- Co-authored-by: Andrew Wilkins <[email protected]> Co-authored-by: Sean Marciniak <[email protected]>
Description
In the current implementation, the BackOff mechanism is trigerred only by memory limiter error. However, the same behavior is expected for the
exporterhelper.ErrQueueIsFullerror, which occurs when there is a sending queue overflow (both memory queue and persistent queue).Link to tracking issue
#39580
Testing
Unit coverage added
Documentation
No documentation updated