azurerm_databricks_workspace_serverless - add resource and data source

[v-yhyeo0202]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

New azurerm_databricks_workspace_serverless resource and data source are added. The resource corresponds to Azure Databricks with serverless compute_mode. This PR is built on top of commits from #31654. As compared to azurerm_databricks_workspace, the custom_properties property is omitted as all the child properties are not allowed when compute_mode is serverless.

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevant documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

https://hashicorp.teamcity.com/buildConfiguration/TF_AzureRM_AZURERM_SERVICE_PUBLIC_DATABRICKS/615695?buildTab=overview

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_databricks_workspace_serverless - add resource and data source

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #31218

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the provider.

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[alexott]

What about naming it azurerm_databricks_serverless_workspace?

[v-yhyeo0202]

What about naming it azurerm_databricks_serverless_workspace?

After some discussions, we decide to use the name azurerm_databricks_workspace_serverless due to the following reasons.

• It aids searching and filtering, the search key "databricks_workspace" will match both azurerm_databricks_workspace and azurerm_databricks_workspace_serverless.
• Easier to set acceptance test prefix, the TEST_PREFIX TestAccDatabricksWorkspace will match both.

[gerrytan]

Thank you @v-yhyeo0202 I haven't done complete review, but I left few concerns I have based on initial look at the design of this new resource.

[gerrytan]

Private endpoint connection is missing, but also not supported in databricks_workspace.

Can you please try the portal experience and determine if we should include private endpoint support in this PR, or as a separate follow up PR.

[v-yhyeo0202]

I try to create private endpoint connections in Databricks workspace networking Azure portal interface as shown above and compared this with the private endpoint connection created through TestAccDatabricksWorkspace_privateLink. The test configurations manage to create private endpoint connection with azurerm_private_endpoint, which I think should be same as the one created through privateEndpointConnection.

Currently the same private endpoint connection creation configurations are applied in TestAccDatabricksWorkspaceServerless_complete. However, after a more detailed check, I cannot find the similar steps to create private endpoint connection for Databricks serverless workspace through Azure portal. The serverless workspace seems to rely on different mechanism to establish private connection than hybrid workspace. I will investigate deeper into Databricks to find out the correct method to create private connection.

[v-yhyeo0202]

I have added TestAccDatabricksWorkspaceServerless_privateLink to validate the private endpoint creation in Databrick serverless workspace. It is tested that private endpoint can be created through the Terraform configuration listed in the test, similar as that created through Azure portal.

[gerrytan]

Please also call out in the PR description that we're intentionally omitting custom_properties because all sub-properties not supported in the serverless. Include a link to the REST API doc: https://learn.microsoft.com/en-us/rest/api/databricks/workspaces/create-or-update?view=rest-databricks-2026-01-01&tabs=HTTP#workspacecustomparameters

[v-yhyeo0202]

The information has been added.

[gerrytan]

Is this ctx re-creation necessary? Normally we should just use the ctx passed from the function param

[v-yhyeo0202]

Initially, I add the ctx re-creation to call defer cancel() with the intention to release the resources associated with the context. However, after checking the code pattern of typed resource, I find that the defer cancel() code is not used. Since ctx is provided through function parameter as you mention, guess the plugin should be able to handle the releasing of resource. The ctx re-creation is removed.

[gerrytan]

While I understand managed_services_cmk_key_vault_id and managed_services_cmk_key_vault_key_id are existing design from databricks_workspace, I don't like copying it into this new resource because it has inherent performance issue when the tenant has large amount of key vault / keys.

Let's check first if this is feasible and get agreement with HC before we proceed.

[v-yhyeo0202]

Acceptance test results

[venkad-intc]

@v-yhyeo0202 @gerrytan Thank you for this.

Looking forward for this resource to be available. Any ETA would be helpful.

[v-yhyeo0202]

@venkad-intc, the current ETA is by the end of March.

[venkad-intc]

@v-yhyeo0202 Are we still on track for this EOM release?

[v-yhyeo0202]

@venkad-intc there will be delay for the release due to ongoing schema design discussion. The new ETA is not decided yet. For temporary workaround, please use azapi_resource to create the resource. Let me know if you encounter issue in doing so.