-
Notifications
You must be signed in to change notification settings - Fork 95
apm: Document sampling.tail.discard_on_write_failure config #1453
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -53,6 +53,10 @@ If a setting is not supported by {{ech}}, you will get an error message when you | |||||||||
Some settings that could break your cluster if set incorrectly are blocklisted. The following settings are generally safe in cloud environments. For detailed information about APM settings, check the [APM documentation](/solutions/observability/apm/configure-apm-server.md). | ||||||||||
:::: | ||||||||||
|
||||||||||
### Version 9.1+ [ec_version_9_1] | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For other version sections, we specify that
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 9.1 will have 2 more configs than 9.0. One mentioned here, another in #1269 . I agree that explicitly mentioning these are new configs on top of 9.0 would be useful. On a side note as a heads-up, before we spend too much time polishing this doc, I'm also thinking removing this doc altogether since it isn't providing much value after being moved from cloud to apm: elastic/apm-server#13602 |
||||||||||
|
||||||||||
`apm-server.sampling.tail.discard_on_write_failure` | ||||||||||
: Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will bypass sampling and always be indexed, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. The default is `false`. | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Re-styling to present tense as per writing guidelines |
||||||||||
|
||||||||||
### Version 8.0+ [ec_version_8_0_3] | ||||||||||
|
||||||||||
|
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -77,6 +77,11 @@ If a setting is not supported on {{ecloud}}, you will get an error message when | |||||||||
Some settings that could break your cluster if set incorrectly are blocklisted. The following settings are generally safe in cloud environments. For detailed information about APM settings, check the [APM documentation](/solutions/observability/apm/configure-apm-server.md). | ||||||||||
:::: | ||||||||||
|
||||||||||
### Version 9.1+ [ec_version_9_1] | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as my previous comment.
Suggested change
|
||||||||||
|
||||||||||
`apm-server.sampling.tail.discard_on_write_failure` | ||||||||||
: Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will bypass sampling and always be indexed, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. The default is `false`. | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Align with previous suggestion |
||||||||||
|
||||||||||
### Version 8.0+ [ec_version_8_0_3] | ||||||||||
|
||||||||||
This stack version removes support for some previously supported settings. These are all of the supported settings for this version: | ||||||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -85,6 +85,18 @@ Policies map trace events to a sample rate. Each policy must specify a sample ra | |||||
| APM Server binary | `sampling.tail.policies` | | ||||||
| Fleet-managed | `Policies` | | ||||||
|
||||||
### Discard On Write Failure [sampling-tail-discard-on-write-failure-ref] | ||||||
|
||||||
Defines the indexing behavior when trace events fail to be written to storage (e.g. when the storage limit is reached). When set to `false`, traces will bypass sampling and always be indexed, significantly increasing the indexing load. When set to `true`, traces will be discarded, there will be data loss potentially resulting in broken traces. The default is `false`. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Align with previous suggestion |
||||||
|
||||||
Default: `false`. (bool) | ||||||
|
||||||
| | | | ||||||
|------------------------------|------------------------------------------| | ||||||
| APM Server binary | `sampling.tail.discard_on_write_failure` | | ||||||
| Fleet-managed (version 9.1+) | `Discard On Write Failure` | | ||||||
|
||||||
|
||||||
### Storage limit [sampling-tail-storage_limit-ref] | ||||||
|
||||||
The amount of storage space allocated for trace events matching tail sampling policies. Caution: Setting this limit higher than the allowed space may cause APM Server to become unhealthy. | ||||||
|
@@ -93,7 +105,7 @@ A value of `0GB` (or equivalent) does not set a concrete limit, but rather allow | |||||
|
||||||
If this is not desired, a concrete `GB` value can be set for the maximum amount of disk used for tail-based sampling. | ||||||
|
||||||
If the configured storage limit is insufficient, it logs "configured limit reached". The event will bypass sampling and will always be indexed when storage limit is reached. | ||||||
If the configured storage limit is insufficient, it logs "configured limit reached". When the storage limit is reached, the event will be indexed or discarded based on the [Discard On Write Failure](#sampling-tail-discard-on-write-failure-ref) configuration. | ||||||
|
||||||
Default: `0GB`. (text) | ||||||
|
||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -146,7 +146,7 @@ Due to [OpenTelemetry tail-based sampling limitations](/solutions/observability/ | |||||
|
||||||
Tail-based sampling (TBS), by definition, requires storing events locally temporarily, such that they can be retrieved and forwarded when a sampling decision is made. | ||||||
|
||||||
In an APM Server implementation, the events are stored temporarily on disk instead of in memory for better scalability. Therefore, it requires local disk storage proportional to the APM event ingestion rate and additional memory to facilitate disk reads and writes. If the [storage limit](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-storage_limit-ref) is insufficient, sampling will be bypassed. | ||||||
In an APM Server implementation, the events are stored temporarily on disk instead of in memory for better scalability. Therefore, it requires local disk storage proportional to the APM event ingestion rate and additional memory to facilitate disk reads and writes. If the [storage limit](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-storage_limit-ref) is insufficient, trace events will be indexed or discarded based on the [discard on write failure](/solutions/observability/apm/tail-based-sampling.md#sampling-tail-discard-on-write-failure-ref) configuration. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I found one other place where the storage limit and sampling bypass was mentioned. Updated to describe the new behavior There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nice catch! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
It is recommended to use fast disks, ideally Solid State Drives (SSD) with high I/O per second (IOPS), when enabling tail-based sampling. Disk throughput and I/O may become performance bottlenecks for tail-based sampling and APM event ingestion overall. Disk writes are proportional to the event ingest rate, while disk reads are proportional to both the event ingest rate and the sampling rate. | ||||||
|
||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This config also applies to
8.19+
but I left it out based on @carsonip comment in another PR. Let me know if I should add8.19+
.