chore(relay): specify spool.enveloppe.max_backpressure_memory_percent configuration for handling relay's failing healthcheck #3635
Conversation
… configuration for handling relay's failing healthcheck Although a fix is being rolled out, that does not mean every relay instance would suddenly be fixed. We would need to still provide a workaround for people to try out. Refer to this specific issue comment: getsentry#3330 (comment)
|
To add more context, which might be helpful for others experiencing the same issue, Relay has two memory settings:
When both are set to 100% (1.0), the system will continue accepting data and the buffer will forward it without issues, but this will eventually cause the process to run out of memory (OOM). As noted before, the buffer will stop producing events downstream only if |
There was a problem hiding this comment.
Rejecting for my comment and mostly for @iambriccardo's comment.
relay/config.example.yml
Outdated
| @@ -14,10 +14,14 @@ processing: | |||
|
|
|||
| # In some cases, relay might fail to find out the actual machine memory | |||
| # therefore it makes the healthcheck fail and events can't be submitted. | |||
| # See https://github.com/getsentry/self-hosted/issues/3330 | |||
| # As a workaround, uncomment the following line: | |||
There was a problem hiding this comment.
Following lines? We should also clearly mark where those lines start and end?
Co-authored-by: Riccardo Busetti <[email protected]>
relay/config.example.yml
Outdated
| # As a workaround, uncomment the following line: | ||
| # | ||
| # health: | ||
| # max_memory_percent: 1.0 | ||
| # spool: | ||
| # envelopes: | ||
| # path: "/your/path" |
There was a problem hiding this comment.
Can we have an example of this path?
There was a problem hiding this comment.
It should just be a path residing on the Docker volume... or since relay doesn't have a persisted volume, probably it's best to use /tmp/foo directory?
There was a problem hiding this comment.
I'm quite unfamiliar with this change, so could you please bear with me and tell me, is it going to be more than 4GB?
I've seen odd situations when using /tmp and filling it with files bigger than 4GB size.
If that's not the case, then correct, /tmp/foo seems right to me.
There was a problem hiding this comment.
Who not mount a volume to relay for this?
There was a problem hiding this comment.
From the docs, the default size is 500MB. We should be fine, no need for another volume. https://docs.sentry.io/product/relay/options/#spooling
Although a fix is being rolled out, that does not mean every relay instance would suddenly be fixed. We would need to still provide a workaround for people to try out. Refer to this specific issue comment: #3330 (comment)
Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.