[Epic] High-Priority Work: Critical Bugs, Infra Gaps & Core Features

[NGA-TRAN]

This epic ticket highlights our key areas of focus. Each checkbox represents a substantial task that may require thoughtful design and scoped planning. If you're interested in working on one, feel free to open a dedicated ticket with all the necessary details. Even if a ticket already exists, you're encouraged to revisit or revise the design as needed—whatever makes the most sense based on your insights.

Documentation

• Review, update and simplify the README to ensure it’s current and helpful for newcomers. It may be outdated or missing key onboarding information. Feel free to expand the documentation—either within the README or elsewhere—for things like this. Update documents when you add or modify things and/or revisit and update them time to time.
• https://github.com/datafusion-contrib/datafusion-distributed/issues/9

Bugs

• Add EXPLAIN tests and fix bug #62
• Upgrade DF and turn the test on Bug in roundtrip serialization and deserialization of physical plan #51
• #63

Test and Core Infrastructure

• Feel free to open additional tickets as you identify new tasks. Below are a few examples to reference and immediate needs. You're welcome to take the lead on any of them, or bring them up during our meetings to raise awareness and help recruit collaborators.
  - [ ] Replace Ambiguous Tuples with Named Structs for Clarity and Type Safety #21
  - [ ] Centralize Configuration Using Global Config with LazyLock or OnceLock #23
  - [ ] use config for batch size and partition group size in distributed_physical_planning #58
  - [ ] Ensure partition groups are always ranges instead of Vec<u64> #59
  - [ ] add roundtrip tests which vary the number of partitions #60
  - [x] Set Up CI/CD Pipeline with Image Builds, Deployment, and TPC-H Validation #18
• in-progress cluster setup
  - [ ] Florent suggested great feedback a lot to improve. May need someone to pair with me/Nga on this
  - [ ] TODO: Likely more tickets will be created for Florent's suggestions and will be prioritized in the right place
• Verify Worker-Level Data Distribution #56
• Simplify TPC-H Validation #64
• Test Explain Analyze #65
• Build a Distributed-DataFusion-CLI #4
• Local Benchmarking #66
• Enable Client-Side Option Passing in ArrowFlightSQL (e.g., Batch Size, Target Partitions) #12
• Implement Worker Shutdown via all_done on Poison Detection #20
• Add Code Coverage Reporting to CI #29
• Optimize Dockerfile Builds for Cross-Compilation and SIMD-Aware Target CPU #24
• Set Up CI/CD Pipeline with Image Builds, Deployment, and TPC-H Validation #18

Core features

• Add Catalog and its API #67
• Planning for well-partitioned data: Currently we distribute data in round robin fashion which are good for random data distribution. For well-partitioned data, we want to map the partitions to the right workers in the first place

Deferred Ideas for Later Phases

• Implement Query Cost Model for Estimated and Actual Metrics #17
• Add Backpressure-Aware Scheduling to Skip Overloaded Workers #25

[gabotechs]

Closing as everything is completed.