azurerm_databricks_workspace: allow hsm key in managed_services_cmk_key_vault_key_id and managed_disk_cmk_key_vault_key_id

[wuxu92]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

extend HSM keys support for managed_services_cmk_key_vault_key_id and managed_disk_cmk_key_vault_key_id. Skip the acctest by default as the cost of HSM test. No need to update the documentation.

changed the validate function from keyVaultValidate.KeyVaultChildID to keyvault.VersionTypeVersioned, because the API requires the key version field otherwise the api errors like CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: InvalidEncryptionEntitiesKeyVaultValue: The keysource is specified as keyvault but keyvault details are not correctly provided

https://learn.microsoft.com/en-us/azure/databricks/security/keys/cmk-managed-services-azure/cmk-hsm-managed-services-azure#step-3-add-a-key-to-a-workspace

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevant documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

--- PASS: TestAccDatabricksWorkspace_managedHSMCMK (2708.71s)
PASS

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_databricks_workspace: allow hsm key in managed_services_cmk_key_vault_key_id and managed_disk_cmk_key_vault_key_id

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #27738

AI Assistance Disclosure

• AI Assisted - This contribution was made by, or with the assistance of, AI/LLMs

Rollback Plan

If a change needs to be reverted, we will publish an updated version of the provider.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[teowa]

Hi @wuxu92, thank you for the PR. I left a small comment.

[teowa]

Since this is tied to a specific subscription, should we consider reading it from environment variables?

[wuxu92]

Good idea, I updated the logic to get the service principal from env var at first.

[magodo]

I've made the change at: https://github.com/hashicorp/terraform-provider-azurerm/pull/31509/changes. So please expect conflict once that PR is merged.

[magodo]

There is a property managed_disk_cmk_rotation_to_latest_version_enabled allows to use the latest cmk version, which brings me the concern about restricting the keys to be versioned.

[wuxu92]

I tested in Portal with a versonless key+rotation enabled, update failed with: Error in enabling customer-managed key encryption. Error: Format of key version used for encryption is invalid. Also tested in Terraform with a versionless key+rotation, the API errors:

{
    "error": {
        "code": "EncryptionKeyVersionFormatInvalid",
        "message": "Format of key version used for encryption is invalid."
    }
}

[teowa]

Thanks @wuxu92 , LGTM!

[wuxu92]

test result:

~/a/a/i/s/databricks ❯❯❯ ARM_TEST_HSM_KEY=1 ./databricks.test -test.run TestAccDatabricksWorkspace_managedHSMCMK -test.v
=== RUN   TestAccDatabricksWorkspace_managedHSMCMK
=== PAUSE TestAccDatabricksWorkspace_managedHSMCMK
=== CONT  TestAccDatabricksWorkspace_managedHSMCMK
--- PASS: TestAccDatabricksWorkspace_managedHSMCMK (2764.54s)
PASS