azurerm_log_analytics_workspace_table Allow deploying of DCR based custom log tables

[oWretch]

Community Note

• Please vote on this PR by adding a 👍 reaction to the original PR to help the community and maintainers prioritize for review
• Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for PR followers and do not help prioritize for review

Description

Enable the creation of DCR-based custom log tables in a Log Analytics workspace.

This PR will break the existing initial workflow of auto-importing a default Microsoft table when changing the retention policy. Instead of auto-importing, the resource will now return an ImportRequired error instead, forcing the inbuilt table to be imported.

To prevent further breakage, I have defined a default for the type property to be Microsoft, though this will cause a drift for any tables that were managed and not a Microsoft table. I have also defined the sub_type to be both Optional and Computed rather than creating a default to again uplift the existing usage of the resource where possible.

I had hoped to get this in before the release of v4.0, but just missed it. So I have conditioned the type and sub_type properties as Optional for 4.0 and Required for 5.0.

PR Checklist

• I have followed the guidelines in our Contributing Documentation.
• I have checked to ensure there aren't other open Pull Requests for the same update/change.
• I have checked if my changes close any open issues. If so please include appropriate closing keywords below.
• I have updated/added Documentation as required written in a helpful and kind way to assist users that may be unfamiliar with the resource / data source.
• I have used a meaningful PR title to help maintainers and other users understand this change and help prevent duplicate work.
  For example: “resource_name_here - description of change e.g. adding property new_property_name_here”

Changes to existing Resource / Data Source

• I have added an explanation of what my changes do and why I'd like you to include them (This may be covered by linking to an issue above, but may benefit from additional explanation).
• I have written new tests for my resource or datasource changes & updated any relevent documentation.
• I have successfully run tests with my changes locally. If not, please provide details on testing challenges that prevented you running the tests.
• (For changes that include a state migration only). I have manually tested the migration path between relevant versions of the provider.

Testing

• My submission includes Test coverage as described in the Contribution Guide and the tests pass. (if this is not possible for any reason, please include details of why you did or could not add test coverage)

TF_ACC=1 go test -v ./internal/services/loganalytics -run="TestAccLogAnalyticsWorkspaceTable_" -timeout 60m -ldflags="-X=github.com/hashicorp/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccLogAnalyticsWorkspaceTable_updateTableRetention
=== PAUSE TestAccLogAnalyticsWorkspaceTable_updateTableRetention
=== RUN   TestAccLogAnalyticsWorkspaceTable_plan
=== PAUSE TestAccLogAnalyticsWorkspaceTable_plan
=== RUN   TestAccLogAnalyticsWorkspaceTable_customDcr
=== PAUSE TestAccLogAnalyticsWorkspaceTable_customDcr
=== CONT  TestAccLogAnalyticsWorkspaceTable_updateTableRetention
=== CONT  TestAccLogAnalyticsWorkspaceTable_customDcr
=== CONT  TestAccLogAnalyticsWorkspaceTable_plan
--- PASS: TestAccLogAnalyticsWorkspaceTable_plan (248.69s)
--- PASS: TestAccLogAnalyticsWorkspaceTable_customDcr (260.43s)
--- PASS: TestAccLogAnalyticsWorkspaceTable_updateTableRetention (265.10s)
PASS
ok      github.com/hashicorp/terraform-provider-azurerm/internal/services/loganalytics  267.917s

Change Log

Below please provide what should go into the changelog (if anything) conforming to the Changelog Format documented here.

• azurerm_log_analytics_workspace_table - allow creating new DCR-based custom log tables [azurerm_log_analytics_workspace_table Allow deploying of DCR based custom log tables #27155]

This is a (please select all that apply):

• Bug Fix
• New Feature (ie adding a service, resource, or data source)
• Enhancement
• Breaking Change

Related Issue(s)

Fixes #21897
Fixes #23359

Note

If this PR changes meaningfully during the course of review please update the title and description as required.

[github-actions]

This PR is being labeled as "stale" because it has not been updated for 30 or more days.

If this PR is still valid, please remove the "stale" label. If this PR is blocked, please add it to the "Blocked" milestone.

If you need some help completing this PR, please leave a comment letting us know. Thank you!

[github-actions]

This PR is being labeled as "stale" because it has not been updated for 30 or more days.

If this PR is still valid, please remove the "stale" label. If this PR is blocked, please add it to the "Blocked" milestone.

If you need some help completing this PR, please leave a comment letting us know. Thank you!

[nclaeys]

@oWretch can you fix the linting errors and rebase? Maybe we can get this branch finally merged 🙏

[oWretch]

Thanks for the review @nclaeys.

As per the PR description, the reason for the linting error is:

To prevent further breakage, I have defined a default for the type property to be Microsoft, though this will cause a drift for any tables that were managed and not a Microsoft table. I have also defined the sub_type to be both Optional and Computed rather than creating a default to again uplift the existing usage of the resource where possible.

I had hoped to get this in before the release of v4.0, but just missed it. So I have conditioned the type and sub_type properties as Optional for 4.0 and Required for 5.0.

If we are happy to accept the plan diff for existing managed resource I can remove the Computed property from the sub_type property. Otherwise I'm open to other options as to how to uplift the existing resources without the linting problem.

[nclaeys]

@katbyte @stephybun can you have a look? I really need this feature to be added to the provider and it has been out there without feedback for months. I want to help where needed to get this merged.

[nunoalima]

If I understood correctly, this PR closes:

• Support for creating a new table in Log Analytics workspace #23359.

[eddy-vera]

Any update on this?

[oWretch]

Thanks for the review @wyattfry. Updates made as requested.

[wyattfry]

Here and further down the explicit check.That(...)s can be omitted as the test automatically will fail if there's a difference between configuration and the resulting resource.

Suggested change

	check.That("azurerm_log_analytics_workspace_table.test").Key("id").Exists(),
	check.That("azurerm_log_analytics_workspace_table.test").Key("name").HasValue("AppEvents"),
	check.That("azurerm_log_analytics_workspace_table.test").Key("retention_in_days").HasValue("7"),

[wyattfry]

Thanks for your patience and work on this so far. I had a couple questions:

1. Could this resource (table) be split into multiple resources instead of adding a type property? i.e. azurerm_log_analytics_workspace_table_microsoft, azurerm_log_analytics_workspace_table_custom_log, azurerm_log_analytics_workspace_table_restored_logs, azurerm_log_analytics_workspace_table_search_results -- then we could programmatically enforce things like custom log tables having names that end with _CL, as well as following the pattern in the rest of the provider and avoiding a breaking change. Or is there a hard constraint that would make this impractical or impossible?
2. I ran the acceptance tests, most passed but TestAccLogAnalyticsWorkspaceTable_customDcr failed part way through with some drift:

------- Stdout: -------
=== RUN   TestAccLogAnalyticsWorkspaceTable_customDcr
=== PAUSE TestAccLogAnalyticsWorkspaceTable_customDcr
=== CONT  TestAccLogAnalyticsWorkspaceTable_customDcr
    testcase.go:173: Step 4/6 error: After applying this test step, the non-refresh plan was not empty.
        stdout:
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        Terraform will perform the following actions:
          # azurerm_log_analytics_workspace_table.test will be updated in-place
          ~ resource "azurerm_log_analytics_workspace_table" "test" {
                id                      = "/subscriptions/*******/resourceGroups/*******/providers/Microsoft.OperationalInsights/workspaces/*******/tables/CustomTable_CL"
                name                    = "CustomTable_CL"
                # (12 unchanged attributes hidden)
              - column {
                  - display_by_default = true -> null
                  - hidden             = false -> null
                  - name               = "CompanyName" -> null
                  - type               = "string" -> null
                    # (3 unchanged attributes hidden)
                }
                # (2 unchanged blocks hidden)
            }
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccLogAnalyticsWorkspaceTable_customDcr (189.18s)

[oWretch]

Thanks @wyattfry. There is no hard requirement for a single resource - I extended the existing resource initially to prevent duplicating extra code. The new table formats have come subsequently, so I'm happy to change track and move to separate resources. It would also solve the 5.0 upgrade differences.

[wyattfry]

Thanks @wyattfry. There is no hard requirement for a single resource - I extended the existing resource initially to prevent duplicating extra code. The new table formats have come subsequently, so I'm happy to change track and move to separate resources. It would also solve the 5.0 upgrade differences.

In that case, let's go with the separate resources approach. I'd be happy to take some of them!

[wyattfry]

I have a draft PR started for the Microsoft type table: #30663

[oWretch]

Thanks @wyattfry. I'll try to get to the custom log resource - life is busy at present so it might be a couple of weeks.

[wyattfry]

Thanks @wyattfry. I'll try to get to the custom log resource - life is busy at present so it might be a couple of weeks.

No worries, I got one started: #30800

[sreallymatt]

closing in favour of the separate resource PRs as mentioned above, #30800 and #31754