Skip to content

[DO NOT MERGE] refactor(spark): Remove spark datasource relations #14063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

[DO NOT MERGE] refactor(spark): Remove spark datasource relations #14063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
72e1840
remove glob paths and deprecate read paths support
Oct 8, 2025
f0ff74d
remove in memory file index test
Oct 8, 2025
46d9e72
try fixing build
Oct 8, 2025
dd865fa
fix some tests
Oct 8, 2025
693e5a1
fix TestDataSourceForBootstrap.scala
Oct 8, 2025
af6fa26
throw exception if fg reader is disabled
Oct 8, 2025
6acf9f5
fix TestPartialUpdateAvroPayload
Oct 8, 2025
33fbf19
resolve or disable more failing tests
Oct 8, 2025
296a6bb
more test fixes and disables
Oct 8, 2025
26bda0c
disable another test
Oct 8, 2025
92db06c
remove test that doesn't make sense to have anymore
Oct 8, 2025
b05812c
disable more tests that try to read the mdt
Oct 8, 2025
5336dfe
disable tests that read from mdt
Oct 8, 2025
82a2fa4
Remove all relations. Keep organization of the incrementals the same …
Oct 8, 2025
53818c2
fix build issues
Oct 9, 2025
adc4e9e
remove legacy parquet file formats. try fixing build
Oct 9, 2025
a8a693a
try get building again
Oct 9, 2025
87b2804
deprecate another config
Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions ...client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,12 @@ public class HoodieBootstrapConfig extends HoodieConfig {
.sinceVersion("0.6.0")
.withDocumentation("Selects the mode in which each file/partition in the bootstrapped dataset gets bootstrapped");

@Deprecated
public static final ConfigProperty<String> DATA_QUERIES_ONLY = ConfigProperty
.key("hoodie.bootstrap.data.queries.only")
.defaultValue("false")
.markAdvanced()
.deprecatedAfter("1.2.0")
.sinceVersion("0.14.0")
.withDocumentation("Improves query performance, but queries cannot use hudi metadata fields");

Expand Down
86 changes: 0 additions & 86 deletions hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/util/PathUtils.scala
View file
Open in desktop

This file was deleted.

6 changes: 0 additions & 6 deletions hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,11 +150,6 @@ trait SparkAdapter extends Serializable {
"hudi".equalsIgnoreCase(provider)
}

/**
* Create instance of [[ParquetFileFormat]]
*/
def createLegacyHoodieParquetFileFormat(appendPartitionValues: Boolean): Option[ParquetFileFormat]

def makeColumnarBatch(vectors: Array[ColumnVector], numRows: Int): ColumnarBatch

/**
Expand All @@ -170,7 +165,6 @@ trait SparkAdapter extends Serializable {
def createRelation(sqlContext: SQLContext,
metaClient: HoodieTableMetaClient,
schema: Schema,
globPaths: Array[StoragePath],
parameters: java.util.Map[String, String]): BaseRelation

/**
Expand Down
3 changes: 1 addition & 2 deletions ...mmon/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,4 @@
# limitations under the License.


org.apache.hudi.BaseDefaultSource
org.apache.spark.sql.execution.datasources.parquet.LegacyHoodieParquetFileFormat
org.apache.hudi.BaseDefaultSource
209 changes: 0 additions & 209 deletions ...rk-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala
View file
Open in desktop

This file was deleted.

2 changes: 2 additions & 0 deletions ...spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ object DataSourceReadOptions {
val REALTIME_PAYLOAD_COMBINE_OPT_VAL = HoodieReaderConfig.REALTIME_PAYLOAD_COMBINE
val REALTIME_MERGE: ConfigProperty[String] = HoodieReaderConfig.MERGE_TYPE

@Deprecated
val READ_PATHS: ConfigProperty[String] = ConfigProperty
.key("hoodie.datasource.read.paths")
.noDefaultValue()
Expand All @@ -88,6 +89,7 @@ object DataSourceReadOptions {
@Deprecated
val READ_PRE_COMBINE_FIELD = HoodieWriteConfig.PRECOMBINE_FIELD_NAME

@Deprecated
val ENABLE_HOODIE_FILE_INDEX: ConfigProperty[Boolean] = ConfigProperty
.key("hoodie.file.index.enable")
.defaultValue(true)
Expand Down
Loading
Loading