Integrate Process Piece Deletions

[ZenGround0]

This is pure AI and can't really even be called a draft. More like an experimental scaffold of spike. I pointed the AI at a document I socratically built up with a lot of effort (I checked it in here if its useful but don't want it to actually merge). I think it might be worth using as a jumping off point but heavy warning that this might be low quality.

[ZenGround0]

Still trying to figure out what to do with this exactly. As it stands there is ambiguity between a dataset moving from 0 to some leaves and a dataset maintaining nonzero leaves but its challenge getting zeroed out because of processDeletes.

Maybe we can schedule this task to only ever run after a nextProvingPeriod has been called?

[ZenGround0]

Ok this is somewhat cleaned up. The most egregious things are removed.

[BigLep]

This is relevant for #1422 right?

[ZenGround0]

A few TODOs to focus on for getting this to a good state:

• Implement SupportsPieceDeletionProcessing correctly and reusing existing version checking code.
• nextProvingPeriod should actually be correctly handling its own deletion processing before the network upgrade
• Make sure nextProvingPeriod “hasDrainInFlight” deferrals actually lead to nextProvingPeriod getting scheduled again.
• Update handler scheduling deletions so it adds a row to the deletion drain table, I think I removed that in error. In particular I don’t think we need to wait for message send with mwe or anything. The table is best effort
  • we will need to do something fairly clever with clearing the table. Specifically we should be checking for inflight schedule removal messages before removing
• Clean up the deletion drain table including thinking more about the name and probably getting rid of the whole blocked_at concept

[ZenGround0]

I'm realizing that the AIs over complexity can be addressed by tracking a need for draining the deletion queue inside the data set data model. I think we want a field "needs_deletion_drain" on dataset

1. Scheduling for proving doesn't change
2. Scheduling for nextProvingPeriod only changes by also including a check that "needs_deletion_drain"
3. processDeletions is scheduled for every data set with "needs_deletion_drain" == true

[ZenGround0]

Closing in favor of #1453