tune automation jitter to reduce tps spikes during rounds

[nicu-da]

We must be careful with the config to not push triggers too much into the future and risk missing rewards

[nicu-da]

so with 300s interval for rewards:

so conclusion:

• we got a 10-15% decrease of peaks by increasing the rewards collection from 200s to 300s
• peaks are between rounds, not the rounds themselves, so it's indeed mostly reward collection
• I don't think it's easy to get more out of this without reducing the chances for rewards collection

[nicu-da]

Actually on second thought here's our current setup:

• interval set to 6m (300s)
• jitter set to 0.5

The way the actual delay between runs gives us a possible running interval between 4.5m and 7.5m

This means that we're actually spreading the bulk of them over the 3m which usually hit right after the round.

What we could do is check the time of the next closing round and schedule a run using a random duration between 1m (to leave a bit of a gap for the round transactions to close the round) and 7-8m after the round is scheduled to close.
This way we would spread them evenly.

If a validator somehow gets an error we still have the retries, and we can schedule another run faster if needed.

wdyt @moritzkiefer-da

[moritzkiefer-da]

This means that we're actually spreading the bulk of them over the 3m which usually hit right after the round.

hm why? They are not synced across nodes are they? So the 4.5m-7.5m don't hit at the same time for all nodes.

[moritzkiefer-da]

What we could do is check the time of the next closing round and schedule a run using a random duration between 1m (to leave a bit of a gap for the round transactions to close the round) and 7-8m after the round is scheduled to close.
This way we would spread them evenly.

I would turn it around: You have a task you want to do in a round. You have 20min to do it. So schedule that task in a random point in the first 15min of that. If you are in the last 5min better just try directly noone should hit that under normal conditions. After you've done one round, look at the timing for the next round and schedule your run using the same procedure.

[nicu-da]

I would turn it around: You have a task you want to do in a round. You have 20min to do it. So schedule that task in a random point in the first 15min of that. If you are in the last 5min better just try directly noone should hit that under normal conditions. After you've done one round, look at the timing for the next round and schedule your run using the same procedure.

Doesn't this mean that now we have a 5 min overlap so the distribution is uneven again and we have a period that spikes? Furthermore this also means that we need to carefully schedule the next run if the first fails because it has to be before the 20m elapses.

This also complicates things because some validators will collect now for one round and others will collect for 2 rounds (though I haven't checked how this is done and if it's batched then we wouldn't care).

[nicu-da]

hm why? They are not synced across nodes are they? So the 4.5m-7.5m don't hit at the same time for all nodes.

No but that's the behavior that we see on the networks as well, in cilr the bulk of transactions runs within a 3-4m interval with the rest taking one or more two minutes where we see descending tps. And it maps out with most triggers being scheduled to run around the same interval after the round closes because of the 4.5m-7,5m interval that limits the interval in the round where you can get scheduled. IRL the earliest you can get scheduled is right after the issuing round is open and the latest is 7.5m afterwards. but statistically it's gonna be somewhere in between

[moritzkiefer-da]

Doesn't this mean that now we have a 5 min overlap so the distribution is uneven again and we have a period that spikes?

I don't think you can fully avoid making it somewhat uneven? You have the periods right before a round ends where you will usually have less activity because it becomes unsafe to wait that long.

Furthermore this also means that we need to carefully schedule the next run if the first fails because it has to be before the 20m elapses.

well if it fails either you just retry immediately through the normal retry logic or you pick a new random time with the same constraint. That seems fine?

This also complicates things because some validators will collect now for one round and others will collect for 2 rounds (though I haven't checked how this is done and if it's batched then we wouldn't care).

Minting activity records is batched. Creating activity records is not. But I'm not sure that is an issue?

[moritzkiefer-da]

Or are you just suggesting to pick it within the first 10min instead of the first 15min? That also sounds fine

[nicu-da]

I don't think you can fully avoid making it somewhat uneven? You have the periods right before a round ends where you will usually have less activity because it becomes unsafe to wait that long.

Why does it matter if we have 2 rounds you can collect the reward? If we always target the 10m right after the issuing round is open we still have plenty of time to hit it next time even with the regular processing just in case.

[moritzkiefer-da]

Right first 10min is probably better

[nicu-da]

after the new scheduling landed in cilr:

before

after

so we got about 10-15% off the peak extra (compared to running with 300s intervals)