Remove legacy DataFrame implementation

[phofl]

Closes #xxxx

• Tests added / passed
• Passes pre-commit run --all-files

sibling of dask/dask#11606

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files   -     2      27 suites   - 2   11h 37m 53s ⏱️ - 38m 6s
 4 127 tests  -     3   4 013 ✅  -     1    112 💤 +  1  2 ❌  - 2 
51 714 runs   - 4 126  49 409 ✅  - 3 997  2 300 💤  - 127  5 ❌  - 1 

For more details on these failures, see this check.

Results for commit 8651e41. ± Comparison against base commit 922888e.

This pull request removes 3 tests.

distributed.shuffle.tests.test_graph ‑ test_basic
distributed.shuffle.tests.test_merge ‑ test_minimal_version
distributed.shuffle.tests.test_shuffle ‑ test_minimal_version

This pull request skips 1 test.

distributed.shuffle.tests.test_shuffle ‑ test_restrictions

♻️ This comment has been updated with latest results.

[hendrikmakait]

Thanks, @phofl. LGM as long as CI is green.