Retry transactions unsent because of too low `maxGasPrice`

[leszko]

Is your feature request related to a problem? Please describe.

Currently, if the value of -maxGasPrice is too low, the transactions are not sent and never retried.

The issue was initially mentioned by @0xVires in the comment.

Would it be possible to also add a retry mechanism like the one for the ticket redemption? Currently, if the new round starts and the gas price is higher than the maxFeePerGas, the O does not call the reward tx. It's then up to the user to manually call it or it won't be called at all.
It would be great if the node would keep checking the gas price and automatically submits the tx as soon as the gas price is lower than the set maxFeePerGas (like for the ticket redemption).

Describe the solution you'd like
Livepeer can keep track of unsent transactions and periodically try to send them again.

Describe alternatives you've considered
One alternative would be to always send the transaction even if -maxGasPrice is too low. However, then in a result, Livepeer may require manual interactions because the pending transactions block any subsequent transactions. For details, please check the following comment.

Additional context
N/A

[leszko]

I think the priority of this lower since we moved to Arbitrum. And Arbitrum Nitro is just around the corner.

@0xVires do you think this feature would still be useful at all? CC: @yondonfu

[0xVires]

I agree. Although at the moment we still have some gas prices spikes every month or so where it would be useful, the spikes will be less frequent and lower with Arbitrum and Ethereum improvements.

[yondonfu]

Agreed. I think it makes sense to icebox for now and revisit if the backlog clears up and/or if gas price spikes become more of a problem.

[rickstaa]

Given that these gas spikes are rare, we can also consider introducing a reward_call_errors metric as an intermediate solution in the monitoring service. It would be triggered whenever the reward call fails, so users can set up a Grafana alert when the transaction fails.

You can check out my draft at #2905, and let me know your thoughts. I'm happy to finalize #2905 if you think it is an excellent addition.

[stronk-dev]

Reward calls also fail every now and then for any RPC error, after which is does not retry. Currently Orchestrators run external call-reward scripts to implement retry behaviour. Would be nice if the Orch would retry reward calls on it's own