fix(spark) CTAS with UNION in Spark 4.x native writer

[Shekharrajak]

Which issue does this PR close?

Closes #3429

Rationale for this change

CTAS with UNION fails in Spark 4.x when the Comet native writer is enabled. The requiresNativeChildren check in CometExecRule was too strict - it required children to be CometNativeExec, but sink operators like CometUnionExec, CometCoalesceExec, CometCollectLimitExec extend CometExec directly while still producing Arrow-formatted data.

What changes are included in this PR?

• Changed the type check from CometNativeExec to CometExec in CometExecRule.convertToComet() to allow all Comet operators that produce Arrow-formatted data as input to the native writer.

How are these changes tested?

Added 6 new test cases in CometParquetWriterSuite:

• Basic UNION write (CTAS style)
• Multiple (3+) DataFrames UNION

[andygrove]

LGTM. I confirmed that some of the tests fail without the fix. Thanks @Shekharrajak

[andygrove]

Test failure:

- parquet write with union and coalesce *** FAILED *** (557 milliseconds)
  99 did not equal 98 Expected 98 rows (49 + 49) (CometParquetWriterSuite.scala:214)

[andygrove]

test failure:

- parquet write with union of structs *** FAILED *** (169 milliseconds)
  0 did not equal 1 Expected exactly one CometNativeWriteExec in the plan, but found 0:
  Execute InsertIntoHadoopFsRelationCommand file:/__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-2f9604b3-87ca-41c1-90d8-1cbed0a42386/output.parquet, false, Parquet, [path=/__w/datafusion-comet/datafusion-comet/spark/target/tmp/spark-2f9604b3-87ca-41c1-90d8-1cbed0a42386/output.parquet], ErrorIfExists, [id, person]
  +- WriteFiles
     +- Union
        :- *(1) Project [1 AS id#1514, [Alice,30] AS person#1515]
        :  +- *(1) Scan OneRowRelation[]
        +- *(2) Project [2 AS id#1516, [Bob,25] AS person#1517]
           +- *(2) Scan OneRowRelation[] (CometParquetWriterSuite.scala:

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

[Shekharrajak]

Please trigger the CI workflow .