load_table_changes_as_spark not allowing to specify response format

[sdemarbre-ores]

Hello,

In the load_table_changes_as_spark , we can't specify which response format is expected. Then, an error is thrown when trying to use it :

HTTPError: 400 Client Error: Bad Request for url: https://xxx.azuredatabricks.net/api/2.0/delta-sharing/metastores/xxx/shares/<share_name>/schemas/<schema_name>/tables/<table_name>/changes?startingVersion=47&endingVersion=48 Response from server: { 'details': [ { '@type': 'type.googleapis.com/google.rpc.ErrorInfo' ,domain': 'data-sharing.databricks.com','metadata': { 'dsError': 'DS_UNSUPPORTED_DELTA_TABLE_FEATURES','historySharingStatusStr': ' ''historyShared:true, ''startVersion: 0.','optionStr': 'For DeletionVectors, use DBR with version 14.1(14.2 for CDF and ''streaming) or higher, or delta-sharing-spark with version ''3.1 or higher, and set option '("responseFormat", "delta") to query the table. Or use delta_sharing python connector ''with version 1.1 or higher. Or ''ask your provider to disable ' 'DeletionVectors with\n'' (ALTER TABLE <table_name> SET ''TBLPROPERTIES '(delta.enableDeletionVectors=false)),\n'' rewrite it without Deletion ''Vectors (REORG TABLE ''<table_name> APPLY(PURGE)).','tableFeatures': 'delta.enableDeletionVectors','versionStr': ' version: 50.'},'reason': 'DS_UNSUPPORTED_DELTA_TABLE_FEATURES'}], 'error_code': 'INVALID_PARAMETER_VALUE'

However, in the load_table_changes_as_pandas, we can specify the format

Thus, I've tried to to directly setup the options in my code, and it worked.

spark.read.format("deltaSharing")
    .option("responseFormat", "delta")
    .option("startingVersion", 47)
    .option("endingVersion", 48) 
    .option("readChangeFeed", "true")
    .load( "<abfss_path_to_config_file>#<share_name>.<schema_name>.<table_name>")

I think the code of the load_table_changes_as_spark should be updated to support this feature