fix: modify CometNativeScan to generate the file partitions without instantiating RDD

[mbutrovich]

Which issue does this PR close?

Partially address #2878.

Rationale for this change

native_datafusion calls inputRDD to get the file lists for each partition, but this is wasteful because it actually instantiates readers that it never uses:

datafusion-comet/spark/src/main/scala/org/apache/spark/sql/comet/CometScanExec.scala

Line 205 in 7282d57

lazy val inputRDD: RDD[InternalRow] = {

There's a TODO to fix this:

datafusion-comet/spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala

Line 146 in 7282d57

// TODO: modify CometNativeScan to generate the file partitions without instantiating RDD.

What changes are included in this PR?

CometScanExec.scala:

• Added getFilePartitions()
• Added createFilePartitionsForBucketedScan()
• Added createFilePartitionsForNonBucketedScan()
• Refactored createBucketedReadRDD() and createReadRDD()

CometNativeScan.scala:

• Replaced inputRDD pattern matching with scan.getFilePartitions()

How are these changes tested?

Existing tests.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 73.68421% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.40%. Comparing base (f09f8af) to head (8443ae2).
⚠️ Report is 759 commits behind head on main.

Files with missing lines	Patch %	Lines
...ala/org/apache/spark/sql/comet/CometScanExec.scala	64.28%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2891      +/-   ##
============================================
+ Coverage     56.12%   59.40%   +3.27%     
- Complexity      976     1367     +391     
============================================
  Files           119      167      +48     
  Lines         11743    15330    +3587     
  Branches       2251     2545     +294     
============================================
+ Hits           6591     9107    +2516     
- Misses         4012     4938     +926     
- Partials       1140     1285     +145     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

Thanks @mbutrovich