-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.CosmosService AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.cosmos:v4-itemIndicates this feature will be shipped as part of V4 release trainIndicates this feature will be shipped as part of V4 release trainneeds-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamWorkflow: This issue needs attention from Azure service team or SDK team
Description
Summary
The Spark Catalog currently supports a subset of Cosmos DB container properties when creating containers via TBLPROPERTIES. This issue tracks adding support for the remaining properties.
Related PR: #48349 (added Vector Embedding Policy support)
Currently Supported
| Property | TBLPROPERTIES Key |
|---|---|
| Partition Key Definition | partitionKeyPath |
| Indexing Policy | indexingPolicy |
| Default TTL | defaultTtlInSeconds |
| Analytical Store TTL | analyticalStoreTtlInSeconds |
| Vector Embedding Policy | vectorEmbeddingPolicy (added in #48349) |
Missing Properties
- Unique Key Policy (
UniqueKeyPolicy) — enforce unique constraints on document fields - Conflict Resolution Policy (
ConflictResolutionPolicy) — configure conflict resolution for multi-region writes - Computed Properties (
ComputedProperty) — define server-side computed columns - Client Encryption Policy (
ClientEncryptionPolicy) — configure field-level encryption - Full Text Policy (
CosmosFullTextPolicy) — configure full text search - Change Feed Policy (
ChangeFeedPolicy) — configure change feed behavior
Implementation Notes
Each property follows the same pattern established by the Vector Embedding Policy PR (#48349):
- Add a constant in
CosmosConstants.scala(TableProperties) - Add a getter in
CosmosContainerProperties.scala - Add JSON bridge methods in
SparkModelBridgeInternal.scala - Wire into container creation in
CosmosCatalogCosmosSDKClient.scalaandCosmosCatalogManagementSDKClient.scala - Add integration tests in
CosmosCatalogITestBase.scala
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.CosmosService AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.cosmos:v4-itemIndicates this feature will be shipped as part of V4 release trainIndicates this feature will be shipped as part of V4 release trainneeds-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamWorkflow: This issue needs attention from Azure service team or SDK team