validator-registrations: ensure safe duty scheduling/execution

[iurii-ssv]

Now that #2130 is merged we are scheduling validator-registration duties for execution every time we get a "fee-recipient updated" event (an EL event),

this result in Validator fee recipient changing much faster (in the next 2 epochs at most, vs the next 10 epochs as it was previously), but also introduces a potential issue of overwhelming SSV node with the sheer amount of validator-registration duties scheduled for execution - this is because:

• 1 validator registration duty is scheduled per 1 fee-recipient change
• additionally/independently, periodically we send registrations for every Validator SSV Operator manages (say 3000 registrations spread out throughout 10 epochs = 300 registrations per epoch = 10 registrations per slot)

if all 3000 validators change fee recipient at the same time - this might cause SSV node to overload temporarily until all the corresponding validator-registration duties are processed/dropped (we have some rate limits in place),

during QA testing of #2130:

• we found no critical issues when updating ~300 fee-recipients at a time (this is the largest number we could test this case with), all other duties were executing successfully, there was no significant increase in CPU/Memory usage - the main resource-usage spike was observed in the bandwidth department (to handle all the p2p messages sent around)
• we did notice some increase in latency for all duty executions though (attestations, aggregations, but other duty types are also probably affected to some extent as well) which might have been more of an issue with stage env than anything else ... but allowing all 3000 validators to change fee-recipients that will trigger 3000 validator-registrations duties might have more dramatic effect

the simplest way to solve this would be to just limit the amount of validator-registrations we are willing to process "in real time" (those resulting from fee-recipient change events) - 300 value we've tested with seems like a reasonable limit

[iurii-ssv]

Some useful notes (on how validator-registrations are handled):

• Ethereum events comes in with an owner on it
• we iterate over all our validators to find those shares that have that owner, lets say we found 100 of them
• we generate 100 validator-registration duties (they will execute in 4 slots from the current one to make sure every node in SSV cluster has had enough time to receive the same event)
• each duty executes, maybe some fail/time-out if there are too many - this is a best effort kind of thing (we won't be able to process 1000+ duties but we will be able to process ~10-100 I think)
• and once the duty is finished we need ~1 epoch after that to broadcast the updated validator-registration to the Ethereum network (we spread requests such that all validators are covered in 32 slot, to avoid any kind of spiky sends)

[mubarek57]

M