Rewards Claiming Failing with Nonce Too Low in v0.5.36

[that-kampe-steak]

Describe the bug
Hello, we've been running orchestrators for some time, and with the latest livepeer releases we've updated to v0.5.35 and the latest, v0.5.36 we've been unable to claim our rewards each round with the call to chain failing with

HTTP Response Error 500: Error calling reward: nonce too low

Here's how we're running the client:

livepeer -network arbitrum-one-mainnet -datadir /home/livepeer/data/ -ethUrl https://arbitrum.node -ethKeystorePath /home/livepeer/keystore/ -ethPassword "$(cat /home/livepeer/keystore/ETH_pass)" -orchestrator -orchSecret "$(cat /home/livepeer/keystore/Orchestrator_secret)" -serviceAddr "addr:8935" -pricePerUnit 900 -pixelsPerUnit 1 -v 6 -monitor -reward=true

We're unable to use the CLI to force the manual claim, and have resorted to calling RewardWithHint via Arbiscan for the moment.

Expected behavior
Claim should happen automatically

OS: ubuntu:18.04

[ybstaked]

now seeing:

E0109 21:47:48.069061      22 orchestrator.go:181] manifestID=00d83c9a-1f21-4ee9-bf73-48dfe09fd53e seqNo=5585 orchSessionID=c6923ca0 clientIP=10.129.127.54 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Error receiving ticket sessionID=c6923ca0 recipientRandHash=bbd6e3b0e0f155b740284a07cada8cb71ea4e3b5d7fd6dcb0f503b86be0e8d35 senderNonce=5: invalid ticket senderNonce sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E nonce=5 highest=6

E0109 21:47:48.069115      22 segment_rpc.go:101] manifestID=00d83c9a-1f21-4ee9-bf73-48dfe09fd53e seqNo=5585 orchSessionID=c6923ca0 clientIP=10.129.127.54 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E error processing payment: invalid ticket senderNonce sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E nonce=5 highest=6

[leszko]

I don't see anything in the v0.5.36 release changelog that could cause it. So I think it may be unrelated to the Livepeer version.

Saying that, when your previous transaction is pending (e.g. gas price set too low), you may experience nounce too low issue.

Could you try if maybe restarting the Livepeer node helps?

[that-kampe-steak]

We've restarted the orchestrator client a few times now, the round the process believes it's attempting to claim with seems to be wrong given the status it outputs, and it's using the last round we were able to automatically claim as the round it sends with it to Arbitrum. Not sure where this is getting set?

HTTP Response Error 500: Error calling reward: execution reverted: current round is not initialized
--

[that-kampe-steak]

This seems likely related: #2661

[leszko]

current round is not initialized

This sounds very weird. Which network do you use, arbitrum-one-mainnet?

[cyberj0g]

This seems likely related: #2661

I don't think so, that's go-ethereum error.

[that-kampe-steak]

current round is not initialized

This sounds very weird. Which network do you use, arbitrum-one-mainnet?

Yes we use arbirtum-one-mainnet

[that-kampe-steak]

We're still seeing these logs with v0.5.37

HTTP Response Error 500: Error calling reward: execution reverted: current round is not initialized

Any ideas?