How to handle rollups when more locators exist for one shard

[bmumdluri]

I am in a situation where there are 900K+ locators for single shard. LocatorFetchRunnable is fetching all locators of shard and keeping them heap which is hogging entire heap. And Rollup could not be processed for other shards and GC is trying to get back memory resulting in more CPU utilization. What would be the ideal way to face scenario which really large shards?

[ChandraAddala]

Have a separate node which will process only that one rollup shard. This should help un-blocking other shards.

At some point we discussed some ideas about adding pagination so that we dont load the entire list to memory but I dont believe that has been implemented.

[bmumdluri]

That's not an acceptable option if there are many large shards.

[baolinCloud]

Blueflood can not rollup when there are 10 Million key in metrics_locator. It costs about 1000s to rollup one shard in one slot of metrics_5m, which shard has 75000 metric locators. I don't know how to solve, the cassandra SSD disk IO persist 90% high.