[Proposal] Make partially_verified_transactions cache more robust

[vicsn]

💥 Proposal

Our partially verified transactions cache is not protected from the following hypothetical scenario:

1. Initiate check_transaction and create the cache_key
2. A dynamically imported Stack gets upgraded, used within check_transaction, and then reverted
3. At the end of check_transaction, partially_verified_transactions gets populated with a stale cache_key.

If the revert happens due to a non-deterministic issue such as hardware failure, I posit that it is unlikely to hit a quorum of validators at the same time.

If the revert happens due to a deterministic issue, I posit that a quorum of validators will halt - as all they can do is rerun atomic_speculate on the same state, and manual intervention + restarts will be required.

So therefore this is low priority at the moment. Some possible solutions are:

• Add a Notify mechanism in check_transaction which can prevent partially_verified_transactions to be populated if Stacks are reverted
• Add a block height to partially_verified_transactions entries, so in the case of any revert we will never again use entries at that block height or lower
• Enable check_transaction to track an immutable map of Stacks

[vicsn]

Also note that update_credits_verifying_keys is unsafe, as it does not update any information references by the TransactionCacheKey. If such an update were to happen during a call to check_transaction, it's possible for the cache to contain an invalid entry. Ideally we get rid of that function in its current form so we don't risk ever using it like that again.