cost-tracker-stats overreporting

[apfitzge]

Problem

• With the activation of 2ry7ygxiYURULZCrypHhveanvP5tzZ4toRwVp89oCNSj (replay verification of cost model) we broke the reporting of cost-tracker-stats - it now reports for all slots by all nodes, not just the leader

Proposed Solution

• Report metrics for non-leader nodes under a different metrics name or add a flag to indicate if the node was leader or not

[apfitzge]

cc @tao-stones

[tao-stones]

every nodes should report exactly same thing for same slot, so that should not distort "per slot" stats, but they can report at different time, that might cause trouble if only group by time, is that the issue you experienced?

[apfitzge]

every nodes should report exactly same thing for same slot, so that should not distort "per slot" stats, but they can report at different time, that might cause trouble if only group by time, is that the issue you experienced?

We have no way to determine a specific node's packing just from the metrics database easily. We were trying to figure out last night if certain nodes were producing shitty blocks, but we cannot because they report for every slot, not just their own.

It's fine to report for all slots, but we should at least have some bool on there that indicates if they were leader or not imo.

[apfitzge]

Probably would have been more accurate to title the issue differently. The issue isn't really over reporting, it's that we cannot filter by leader

[tao-stones]

Because not all leaders report stats, with Replay also reporting, metrics should have cost stats for all slots, which is helpful in some cases, however there will be many (identical) duplications for same slot. It should not matter if we want to see how each block looks like, but if we want to look particular host's leader slots, we may have to include leader schedule in query for now (could use replay_stage-my_leader_slot or cli output); For fix, I like more the suggestion that to report cost tracker stats into two tables, one for banking_stage, one for replay_stage.