Core: Support IncrementalChangelogScan with deletes

[manuzhang]

Pick up #6182 with an alternative implementation

cc @aokolnychyi @szehon-ho @flyrain for review

[wypoon]

You have made changes in BaseIncrementalChangelogScan that produces BaseDeletedRowsScanTask in certain cases. However, you have not made changes in the Spark ChangelogRowReader to handle DeletedRowsScanTasks. As reading the changelog has only been implemented in Spark thus far, the changes cannot be consumed.
You have added some tests to the Spark TestCreateChangelogViewProcedure, which actually makes use of ChangelogRowReader to read the changelog, but those tests currently don't actually test any DeletedRowsScanTasks, because the tables used in the test are not created with write modes set to merge-on-read (the default is copy-on-write); no delete files are ever created in the tests.

I took my changes in #10935 and swapped out my BaseIncrementalChangelogScan implementation for yours (see #10954). I was interested to see how your BaseIncrementalChangelogScan implementation works. (I needed my other changes in order for Spark changelog reading to actually work.) I found that testDataFilters and testDataRewritesAreIgnored in TestChangelogTable failed for the merge-on-read case (in the former, the delete is not detected, and in the latter, the second delete is detected but not the first). In TestChangelogReader, testPositionDeletes, testEqualityDeletes and testMixOfPositionAndEqualityDeletes failed. It seems to me that your implementation is not catching all deletes that can happen.

[wypoon]

That is one way! ;-)
In my implementation, I use a placeholder DummyChangelogScanTask that is filtered out.

[manuzhang]

Closed in favor of #10935