[Converter] Refactor converter tests into integration tests and unit tests

[Zyiqin-Miranda]

Summary

This PR mainly contains the following changes:

1. Refactor the converter tests into integration tests and unit tests. Integration tests are tests require Spark setup to ensure position delete result compatibility with Spark. Unit test are mainly only tests for specific functions correctness.
2. Remove broken Iceberg integration tests.
3. Specific cases newly added to ensure position delete produced twice for same data files can correctly read by Spark since we currently always produce position delete for the whole table. The "redundant" position delete files doesn't take into any effect.
4. Stress test for dedupe correctness. Added cross-file and intra-file records (>1000) for multiple files (5-10) containing both cross-file and intra-file duplicates, added assertion for dedupe correctness.

Rationale

Explain the reasoning behind the changes and their benefits to the project.

Changes

List the major changes made in this pull request.

Impact

Discuss any potential impacts the changes may have on existing functionalities.

Testing

Describe how the changes have been tested, including both automated and manual testing strategies.
If this is a bugfix, explain how the fix has been tested to ensure the bug is resolved without introducing new issues.

Regression Risk

If this is a bugfix, assess the risk of regression caused by this fix and steps taken to mitigate it.

Checklist

• Unit tests covering the changes have been added

  • If this is a bugfix, regression tests have been added

• E2E testing has been performed

Additional Notes

Any additional information or context relevant to this PR.

[pdames]

LGTM! Thanks for the delete converter code hardening, and (large!) refactor to separate integ/unit tests.