Some validators are proposing empty blocks on mocha

[evan-forbes]

We're seeing some validators still propose some empty blocks on mocha. This is unexpected since as far as we know some of these nodes are upgraded.

[rach-id]

I was taking a look on this and found out:

• There is no pattern between validators who propose empty blocks. Every time a different one.
• This also started happening on mainnet couple hours ago

So since we manage to spam the network with transactions and all of them get included, I suspect this is not an issue with the mempool not propagating the transactions fast enough, or the consensus reactor not including transactions in the block, but the fact that the mempool is working as expected.
Previously, we had issues with propagating/including transactions from the same account that have different sequence numbers. So, only a small subset of the peers have all the transactions, those who managed to have all the transactions in order. This made all the validators propose a block with a few transactions.
Now, since we propagate the transactions correctly, and we don't have much volume on mocha/mainnet, whenever a client submits a bunch of transactions, they just get included in the next block and that's it.

One way to test this theory is to run the latency tool on mocha with relatively small transactions and only a few transactions, then see how fast they get included.

What do you think?