[EPIC] Improve Cluster Observability

[milenkovicm]

Two major directions (as of now):

Collect Executor Statistics

As starting point we could start by collecting and aggregating executor statistic, something similar to spark UI executor tab

some statistics, such as memory utilisation, disk usage, would be collected on the executor, other like shuffle read and write may be collected on scheduler side.

We would need to expose additional rest interface to expose collected metrics

Per Stage Flame Graph

Similar to Nvidia RAPIDS Per Stage Flame Graph collect stats and produce flame graphs.

We would need to further investigate what should be done

[milenkovicm]

@danielhumanmod interested maybe?

[mohit7705]

Hi, @milenkovicm
I’m interested in contributing to this EPIC. I’d like to start with executor memory usage metrics. Could you please guide me on where to begin?
Thanks!

[danielhumanmod]

@danielhumanmod interested maybe?

Thanks for looping me in - yes I am interested. I can start by doing an initial investigation around the implementation, and draft the epic breakdown. So that we can split into concrete task if anyone want to collaborate.