fix(pdpv0): re-enable DELETE in processPendingCleanup to stop RPC flood (#1033)

processPendingCleanup runs on every Filecoin block and calls PieceLive()
for every piece with removed=TRUE in pdp_data_set_pieces. The DELETE that
would remove confirmed-dead pieces from the table was commented out, causing
the list to grow without bound and flood Lotus with EthCalls.

After the b7a8796 fix (lo.Contains pointer comparison bug), pieces are now
correctly marked removed=TRUE for the first time. On calibration nodes with
~2,900 such pieces, this caused ~2,900 PieceLive() EthCalls per block
(~30s interval), overwhelming Lotus RPC and causing i/o timeouts on all
subsequent eth_estimateGas calls. This in turn caused proving tasks to fail,
and after 5 consecutive failures datasets were marked unrecoverable.

Re-enable the DELETE with an AND removed=TRUE guard for safety. PieceLive()
is called immediately before the DELETE and confirms the piece is gone
on-chain, making this safe. The list drains to zero over a few blocks and
the EthCall flood stops permanently.

Author: TippyFlitsUK

tasks/pdpv0/watch_piece_delete.go

@@ -177,12 +177,11 @@ func processPendingCleanup(ctx context.Context, db *harmonydb.DB, ethClient *eth
 		}
 
 		if !live {
-			// XXX(Kubuxu): commented out as this has lead to proving failures
-			//_, err := db.Exec(ctx, `DELETE FROM pdp_data_set_pieces WHERE data_set = $1 AND piece_id = $2`, piece.DataSetID, piece.PieceID)
-			err = nil
+			_, err = db.Exec(ctx, `DELETE FROM pdp_data_set_pieces WHERE data_set = $1 AND piece_id = $2 AND removed = TRUE`, piece.DataSetID, piece.PieceID)
 			if err != nil {
 				return xerrors.Errorf("failed to delete piece %d: %w", piece.PieceID, err)
 			}
+			log.Infow("deleted confirmed-removed piece from DB", "dataSetId", piece.DataSetID, "pieceId", piece.PieceID)
 		}
 	}