perf: [iceberg] per partition file scan task serialization #4638

Job	Run time
Lint	39s
Build Native Library (macOS)	25m 32s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [expressions]	28m 3s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [csv]	9m 55s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [fuzz]	43m 52s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [shuffle]	23m 22s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [exec]	21m 27s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [parquet]	21m 19s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [csv]	12m 39s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [sql]	8m 12s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [exec]	23m 30s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [expressions]	21m 5s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [sql]	11m 52s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [parquet]	24m 48s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [fuzz]	28m 6s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [fuzz]	23m 38s
macos-14/Spark 3.4, JDK 11, Scala 2.12 [shuffle]	23m 1s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [shuffle]	36m 32s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [parquet]	21m 18s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [exec]	19m 4s
macos-14/Spark 3.5, JDK 17, Scala 2.13 [expressions]	24m 30s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [csv]	9m 39s
macos-14/Spark 4.0, JDK 17, Scala 2.13 [sql]	12m 55s
	7h 54m 58s

Provide feedback