Skip to content

Update Cassandra example config#1120

Merged
dhoard merged 1 commit intoprometheus:mainfrom
russss:cassandra-config
Jan 16, 2025
Merged

Update Cassandra example config#1120
dhoard merged 1 commit intoprometheus:mainfrom
russss:cassandra-config

Conversation

@russss
Copy link
Contributor

@russss russss commented Jan 16, 2025

Cassandra has added a lot more metrics recently. The existing example config results in a scrape time of 15-30 seconds and more than doubles Cassandra's CPU usage in our setup.

This changes the example config to use a whitelist approach with a sample whitelist which takes a much more reasonable 200 ms to scrape.

Cassandra has added a lot more metrics recently. The existing example
config results in a scrape time of 15-30 seconds and more than doubles
Cassandra's CPU usage in our setup.

This changes the example config to use a whitelist approach with a
sample whitelist which takes a much more reasonable 200 ms to scrape.

Signed-off-by: Russ Garrett <russ@garrett.co.uk>
@dhoard
Copy link
Collaborator

dhoard commented Jan 16, 2025

@russss Thanks for the PR!!! Given the other Cassandra example is valid, I feel we should add this as another example.

Thoughts?

@russss
Copy link
Contributor Author

russss commented Jan 16, 2025

Personally, I don't think there's any use for the previous config any more. It's definitely too slow to use with a 15 second prometheus scrape interval, it causes Cassandra to use a lot of CPU, and it generates literally thousands of metrics which will blow up the size of the TSDB.

This will presumably only get worse as Cassandra adds even more metrics, so I think using the blacklist approach is a bad idea in general as you risk Cassandra upgrades flooding your TSDB with new metrics (which is what happened to me).

I think the only reasonable way to use jmx_exporter with Cassandra now is to cherry-pick the metrics you want to use, and I think this is a good starting point as it provides a number of example match patterns.

@russss
Copy link
Contributor Author

russss commented Jan 16, 2025

For reference, I counted the number of metrics on our Cassandra 5.0.2 cluster, which only has a handful of tables.

By my count the unfiltered org.cassandra.metrics namespace would generate 30,753 separate metrics of which 4,078 are histograms with multiple labels.

If you apply the filtering in the existing cassandra.yaml, that reduces it to 15,821 metrics (2,588 histograms).

My proposed change generates 68 metrics of which about 10 are histograms.

@dhoard dhoard merged commit 23808d0 into prometheus:main Jan 16, 2025
2 checks passed
@russss russss deleted the cassandra-config branch January 17, 2025 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants