Reaper pausing on too many compaction > 20

[dasmohapatrak]

Hi,
On our Cassandra cluster, i see the below message no of times when the reaper is running.

SegmentRunner - SegmentRunner declined to repair segment 556369 because of too many pending compactions (> 20) on host.

But when i run the "nodetool compactionstats" on that node, most of the time, i see only one compaction task is running. Though the pending tasks show more than 20 .
Now, i understand the "pending tasks" is actually only an estimate, not the actual no. of tasks pending.

So my question is, why should Reaper wait on the pending tasks, even though there are less than 20 pending tasks running on that node.

Thanks

Kishore

[deniszh]

IIRC "pending tasks" includes all tasks, not only compaction tasks.

[Bj0rnen]

It's the responsibility of the compaction strategy to estimate how many remaining compaction tasks it has, in effect, got "queued up". This, for all tables summed up is what you see as "pending tasks".

If we would go ahead and repair just because there are few ongoing repairs right now, there would be more SSTables piling up, and the node would fall behind on compactions. The pending compactions are what matters, because we don't want to overwhelm the nodes with SSTables.

[dasmohapatrak]

When you mention all tables, does that mean it includes the keyspaces for OpsCenter, system_traces and system also.
We do not run reaper against these keyspaces.

Thanks

Kishore Mohapatra
Principal Database Administrator
Seattle, WA
Ph : 425-691-6417 (cell)
Email : kishore.mohapatra@nuance.commailto:kishore.mohapatra@nuance.com

From: Björn Hegerfors [mailto:notifications@github.com]
Sent: Monday, May 09, 2016 9:25 AM
To: spotify/cassandra-reaper
Cc: Mohapatra, Kishore; Author
Subject: [EXTERNAL] Re: [spotify/cassandra-reaper] Reaper pausing on too many compaction > 20 (#139)

It's the responsibility of the compaction strategy to estimate how many remaining compaction taskshttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_cassandra_blob_e16d8a7a667d50271a183a95be894126cb2a5414_src_java_org_apache_cassandra_db_compaction_AbstractCompactionStrategy.java-23L209&d=CwMCaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=0OeAb06DF5CHvF-hfe4C1baanC0t7TODYltmJ-KNdwQ&s=_f_MdZ7EZIBN1sXpG7vbgn_7pQAxxMx-rPGC4xozEMI&e= it has, in effect, got "queued up". This, for all tables summed up is what you see as "pending tasks".

If we would go ahead and repair just because there are few ongoing repairs right now, there would be more SSTables piling up, and the node would fall behind on compactions. The pending compactions are what matters, because we don't want to overwhelm the nodes with SSTables.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spotify_cassandra-2Dreaper_issues_139-23issuecomment-2D217914486&d=CwMCaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=0OeAb06DF5CHvF-hfe4C1baanC0t7TODYltmJ-KNdwQ&s=xdWVKg4ccyYzrIG07_VfnoFKzr25ijKvQ-Zf-9qIepQ&e=

[Bj0rnen]

I don't actually know. In practice, at least the tables in system and system_traces don't seem to make a meaningful impact, if any at all. It's probably because they are all very small tables (except hints sometimes), orders of magnitude smaller than user tables. Compactions take seconds and don't happen often. Or maybe they aren't even tracked. I don't have any experience with OpsCenter, so I can't give you an answer on that.