feat: Create a new configuration when the Kafka version is changed to avoid ConflictException: A resource with this name already exists errors

[the-middle]

Description

I've tried to upgrade MSK cluster that was created by this module and got an error resource with this name already exists. After checking source files, I found that random_id is not recreated on every update of the configuration or changing version of the MSK cluster.

You can read about this here: https://registry.terraform.io/providers/hashicorp/random/latest/docs

As noted above, the random resources generate randomness only when they are created; the results produced are stored in the Terraform state and re-used until the inputs change, prompting the resource to be recreated.

The resources all provide a map argument called keepers that can be populated with arbitrary key/value pairs that should be selected such that they remain the same until new random values are desired.

Motivation and Context

Seamless experience using this module while upgrading MSK cluster version or changing configuration parameters.

Error: creating MSK Configuration: operation error Kafka: CreateConfiguration, https response error StatusCode: 409, ConflictException: A resource with this name already exists.

Breaking Changes

No breaking changes.

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have tested it on my dev environment

• I have executed pre-commit run -a on my pull request

pre-commit run -a
[INFO] Initializing environment for https://github.com/antonbabenko/pre-commit-terraform.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Terraform fmt............................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
check for merge conflicts................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed

[the-middle]

On the 1 and 2 screenshot you can see what terraform apply is doing. Because I'm changing MSK engine version it needs to recreate configuration with new engine version.
Because of this, it wants to create resource before deleting old one.
lifecycle { create_before_destroy = true }
But it cannot be recreated because old configuration has the same name.

From documentation we know that random_id generate new values only when created. Now there is no way for this resource to be recreated in order to generate new value. So I added keepers to trigger random_id resource recreation on kafka_version or server_properties change.

[bryantbiggs]

why do we need to have server_properties trigger a replacement as well? From what you shared, I think having the version trigger a replacement sounds correct, but I am not seeing where server_properties is necessary

[the-middle]

I had issues with revisions not updating the actual cluster properties. Recreate works all the time and causes no downtime. If you have different experiences, please share

[bryantbiggs]

I don't, which is why I was asking. looking around I see other issues related to this hashicorp/terraform-provider-aws#15795 but not sure what the appropriate course of action is here

[the-middle]

I think, I can change this PR to be cluster version upgrade only and create separate PR for recreating configuration if server_properties was changed

[the-middle]

@bryantbiggs So, do I need to change this PR to be only related to changing kafka_version and create separate for the configuration changing?

[bryantbiggs]

yes - lets only do the kafka_version for now. I don't know that this is valid, seems like there is a race condition on the service (MSK) side, so I don't want to get heavy handed with the hacky updates here

[antonbabenko]

This PR is included in version 2.9.0 🎉

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.