Revisit default TBS storage size limit `sampling.tail.storage_limit` and storage limit handling

[carsonip]

sampling.tail.storage_limit has a default of 3GB which is usually insufficient for high throughput use cases. A storage limit too low will cause TBS to be bypassed and other downstream effects. Reconsider the default in 9.0 release.

Tasks

• TBS: set default sampling.tail.storage_limit to 0 but limit disk usage to 90% #15467
• apm: change tail_sampling_storage_limit config default to 0GB integrations#12543
• cloud: https://elasticco.atlassian.net/browse/CP-10146
• apm: change default sampling.tail.storage_limit observability-docs#4814
• apm: change default sampling.tail.storage_limit docs-content#581
• Follow-up to #581 docs-content#582
• TBS: change default disk usage threshold to 0.8 #15524

[carsonip]

I have 2 ideas for the new default.

• to allow infinite disk usage
• to detect available disk space (on startup / periodically) and set the limit dynamically

The argument for either of them is that e.g. if you are running a mysql server, you expect it to use storage space but you don't expect yourself updating a "storage limit" every time it gets hit.

The argument for a detection is to avoid causing a node to be unhealthy in any environments.

[simitt]

@carsonip the solution to detect available disk space would be more elegant. We would need to ensure that it works on all supported OS systems + docker and have a sensible fallback if disk size cannot be detected.
Do you have capacity to look into this?

[carsonip]

Do you have capacity to look into this?

Yes, it is in scope of this task. I'll write down some possible solutions soon.

[carsonip]

This task highly depends on outcome of #15235

Assuming #15235 goes well, the pebble storage usage profile would be very different from badger. It should be much more predictable, and with that we will be able to come up with either

1. keep storage limit handling, but set a much more reasonable and usable default, and documentation around how to set a good storage limit.
2. (on top of point (1) above) default to disk space detection but fallback to a constant default. There may be complexity in a cross-platform solution.

@simitt what do you think of the other alternatives, are they no-go?
3. keep storage limit handling but default to unlimited (and maybe always set it to a number on ESS)
4. removing storage limit handling altogether. This will simplify #15235.

[simitt]

3. keep storage limit handling but default to unlimited (and maybe always set it to a number on ESS)
4. removing storage limit handling altogether. This will simplify TBS: Replace badger with pebble #15235.

These options do not sound like customer friendly options to me.
Option 1 and 2 both sound reasonable. If option 2 is risky or very complex, I wouldn't see a big problem with starting with option 1 and then potentially switching to option 2 later.

[carsonip]

Option 1 is the baseline. It should be trivial.
I'm also optimistic about option 2 as we move to pebble in #15246 , because there seems to be a cross-platform disk usage stats (available, used, total) in pebble https://github.com/cockroachdb/pebble/blob/master/vfs/disk_usage_linux.go