Misleading titles and descriptions for Tombstones panels in Grafana

[igdrscylladb]

Installation details

Panel Name: Tombstones Writes, Range Tombstones Writes, Cells Tombstones writes
Dashboard Name: Detailed
Scylla-Monitoring Version:
Scylla Monitoring version - 4.8.3
Scylla-Version: 2024.1.13

Request

The tombstones panels presented in the detailed dashboard and within the tombstones section are misleading and bring a lot of confusion when trying to debug issues related to tombstones generation.

Currently these panel are showing the tombstones found in sstables right after a stream/compaction/flush has happened, so it is an aggregated figure not related to any deletion/update of data that may be happening at a moment in time.

First of all the names and descriptions for the tombstones panels mentioned are misleading.
Second there are not metrics about tombstones being generated at a moment in time due to delete/updates itself.

The confusion that bring these panels is about why if pushing a low number of delete/update to my database (or none), I still may see a (huge) number of "tombstones writes" in this misleading panel, believing these are new tombstones being generated when in fact are tombstones remaining after compaction and not expired as still within GC GRACE SECONDS threshold.

Below a tiny reproduction of this issue after manually flushing the tables, tombstones writes reported but not necessarily during deleting/updating records.

[amnonh]

@igdrscylladb, thanks for opening the issue. Do you have an alternative that you think will be clearer regarding what we have now?

[igdrscylladb]

@amnonh

The most immediate fix to this is to rename those tombstones panels e.g. sstables tombstones and give it a more precise descriptions about how/when this data is captured. (this is basically the aim of this GH issue opened)

Regarding presenting metrics about tombstones being generated in (near) real time, that is another beast as for one single deletion/update tombstone should be generated in the table itself but also as part of a propagation if applicable, in a potential index, MW or even cache. I am not sure how this can be captured. If we attend just SSTABLES analysis, then we need to assess these and use the marked deleted/ local deleted timestamps so anything e.g. now() - 60s will give us the new tombstones created in the last minute. At least this is what I would do in a manual way to get information about this.

[amnonh]

@igdrscylladb I'm asking if you can suggest concrete texts

[amnonh]

@igdrscylladb I'm assigning to you to get the info, not for implmentation

[igdrscylladb]

Hi @amnonh sorry, about a more concrete texts you are asking for, not sure if you are referring to panel's titles or descriptions or both. As mentioned, these 3 panels titles are currently misleading so the ask if to change the titles at least.

Furthermore, I am not sure what exactly can be used as a titled for these metrics but my understanding is that the metrics capture tombstones found in SSTABLES after compaction so you can rename them something along those lines.