[BUG] V2 write recognition fails with Iceberg on extraClassPath

[gerashegalov]

Describe the bug
In deployments where the Iceberg runtime is supplied through extraClassPath and loaded by the AppClassLoader, V2 write recognition can fail because GpuSparkWriteAccess is placed under spark-shared inside the cudf-spark jar and is defined across a different classloader boundary.

The cudf-spark jar itself does not need to be deployed through Spark's MutableURLClassLoader; it may also be supplied through extraClassPath. The problem is the internal spark-shared placement of the affected class. In the observed failure, SparkWrite was defined by the AppClassLoader while the spark-shared GpuSparkWriteAccess was defined by org.apache.spark.util.MutableURLClassLoader.

The JVM does not consider classes with the same package name to be in the same runtime package when their defining classloaders differ. As a result, GpuSparkWriteAccess.supports cannot access Iceberg's package-private SparkWrite class and throws an IllegalAccessError.

This can affect non-Iceberg V2 writes. The recognizer added by #13314 evaluates every enabled V2 write recognizer, so a noop write can invoke the Iceberg recognizer even though the write is not an Iceberg write.

Related to #15821. PR #15827 changes Iceberg class placement within the plugin jar, but should not be considered a fix until this split-classloader deployment passes the focused tests.

Steps/Code to reproduce bug

1. Supply the Iceberg Spark runtime using driver and executor extraClassPath, causing Iceberg classes such as org.apache.iceberg.spark.source.SparkWrite to be loaded by the AppClassLoader.
2. Use a cudf-spark jar whose assembled layout places org/apache/iceberg/spark/source/GpuSparkWriteAccess.class under spark-shared. The cudf-spark jar may itself also be supplied through extraClassPath.
3. Enable Iceberg integration and run a non-Iceberg V2 noop write, for example:

spark.range(10).write.format("noop").mode("append").save()

The write fails during plan tagging with an exception similar to:

java.lang.IllegalAccessError: failed to access class
org.apache.iceberg.spark.source.SparkWrite from class
org.apache.iceberg.spark.source.GpuSparkWriteAccess
(SparkWrite is loaded by the AppClassLoader; GpuSparkWriteAccess is loaded by
org.apache.spark.util.MutableURLClassLoader)

The focused noop_write_test.py append, overwrite, consumes-input, and evaluates-input cases exercise the failure.

Expected behavior
Non-Iceberg V2 writes must not fail while probing the Iceberg recognizer. Recognition should either avoid loading package-private Iceberg implementation classes for unrelated writes or safely handle the split-classloader deployment.

Iceberg writes should retain package-private access when the Iceberg runtime is supplied through extraClassPath.

Environment details (please complete the following information)

• Environment location: Any deployment where the Iceberg runtime is supplied through driver/executor extraClassPath
• Spark configuration settings related to the issue: Iceberg runtime on spark.driver.extraClassPath and spark.executor.extraClassPath; cudf-spark may also be on extraClassPath; GpuSparkWriteAccess is packaged under spark-shared

Additional context
#14866 introduced GpuSparkWriteAccess to bridge Iceberg package-private APIs across the shim layout. #13314 made the failure visible to unrelated V2 writes by evaluating the Iceberg recognizer while recognizing noop writes.

Potential fixes include keeping the package-private Iceberg access bridge out of spark-shared and ensuring it is defined by the same classloader as the Iceberg runtime. Short-circuiting recognition once the noop recognizer matches, or making the Iceberg recognizer avoid package-private class linkage during probing, would additionally prevent unrelated V2 writes from exposing the Iceberg classloader problem. The four noop-write tests should be used as acceptance coverage with Iceberg and, separately, both Iceberg and cudf-spark on extraClassPath.