Skip to content

[Cosmos Spark] Add missing container property support in Spark Catalog #48360

@xinlian12

Description

@xinlian12

Summary

The Spark Catalog currently supports a subset of Cosmos DB container properties when creating containers via TBLPROPERTIES. This issue tracks adding support for the remaining properties.

Related PR: #48349 (added Vector Embedding Policy support)

Currently Supported

Property TBLPROPERTIES Key
Partition Key Definition partitionKeyPath
Indexing Policy indexingPolicy
Default TTL defaultTtlInSeconds
Analytical Store TTL analyticalStoreTtlInSeconds
Vector Embedding Policy vectorEmbeddingPolicy (added in #48349)

Missing Properties

  • Unique Key Policy (UniqueKeyPolicy) — enforce unique constraints on document fields
  • Conflict Resolution Policy (ConflictResolutionPolicy) — configure conflict resolution for multi-region writes
  • Computed Properties (ComputedProperty) — define server-side computed columns
  • Client Encryption Policy (ClientEncryptionPolicy) — configure field-level encryption
  • Full Text Policy (CosmosFullTextPolicy) — configure full text search
  • Change Feed Policy (ChangeFeedPolicy) — configure change feed behavior

Implementation Notes

Each property follows the same pattern established by the Vector Embedding Policy PR (#48349):

  1. Add a constant in CosmosConstants.scala (TableProperties)
  2. Add a getter in CosmosContainerProperties.scala
  3. Add JSON bridge methods in SparkModelBridgeInternal.scala
  4. Wire into container creation in CosmosCatalogCosmosSDKClient.scala and CosmosCatalogManagementSDKClient.scala
  5. Add integration tests in CosmosCatalogITestBase.scala

Metadata

Metadata

Assignees

Labels

ClientThis issue points to a problem in the data-plane of the library.CosmosService AttentionWorkflow: This issue is responsible by Azure service team.cosmos:v4-itemIndicates this feature will be shipped as part of V4 release trainneeds-team-attentionWorkflow: This issue needs attention from Azure service team or SDK team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions