Check EOS issue #4649 speculation node performance

[abitmore]

Do we have the same issue? EOSIO/eos#4649

And this: EOSIO/eos#4706

It's related to re-validating pending transactions after pushed new blocks.

[jmjatlanta]

My interpretation of the above:

As it works today:
My node is stale, and busy syncing. It receives a transaction (not part of a block). It applies the transaction as pending, and then has to hastle with it as new blocks come in (roll it back, apply block, see if it was part of the block, and if not, apply it again).

The way it could work:
If we are stale, ignore incoming transactions that are not part of a block. Existing pending transactions would be dealt with as normal.

If my interpretation is correct, should something be done with transactions received while it was syncing that were not part of a block? Or would it be better to just ignore them and let the later blocks catch me back up?

If I must store the transactions to apply after the sync, I now need to research if they have already been received within a (recently acquired) block. In extreme cases, that has the potential to cause me to go stale again. It would be nice to throw them out if possible.

[abitmore]

A scenario that need to consider is, a node can become stale when it was in sync, so in this case it's dangerous to simply throw away pending transactions. Also, why the node became stale does matter.

In regards to ignoring or rejecting new transactions, I guess we need to review p2p logic, we should not disconnect the peer when rejecting the transaction due to this reason, nor propagate the transaction if it's not validated by the node.

The worst thing is we don't know if a node is really stale. Head block time can be a bit far in the past when too many witnesses are offline due to whatever reason. It's important for block producing nodes to keep pending transactions so they can add them to new blocks.

[pmconrad]

We need to revisit the pending transactions logic in any case.
Current behaviour can confuse clients and lead to node overload.

[abitmore]

Steem implemented something related to this issue some months ago: steemit/steem#3333 .
TL;DR: keep a limited number of pending transactions in memory, drop others.