CLI-3508: Persistent terraform drift when creating a confluent_schema that already existed as one of the previous versions #619
Description
What
There's a bug in the TF Provider that results in persistent Terraform drift when creating a confluent_schema that already existed as one of the previous versions. In other words, if a schema evolves into something that matches an old version of the schema, it won't get a new version. Instead, a user will experience persistent Terraform drift:
✗ terraform apply -auto-approve ...
...
Plan: 0 to add, 1 to change, 0 to destroy.
confluent_schema.key: Modifying... [id=lsrc-3mndmj/enrichment-key-test-value/latest]
confluent_schema.key: Still modifying... [id=lsrc-3mndmj/enrichment-key-test-value/latest, 10s elapsed]
confluent_schema.key: Modifications complete after 11s [id=lsrc-3mndmj/enrichment-key-test-value/latest]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
✗ terraform plan
...
~ update in-place
Terraform will perform the following actions:
# confluent_schema.key will be updated in-place
~ resource "confluent_schema" "key" {
id = "lsrc-3mndmj/enrichment-key-test-value/latest"
~ schema = jsonencode(
~ {
~ fields = [
{
name = "code"
type = "string"
},
- {
- name = "code2"
- type = "string"
},
]
name = "EnrichmentKey"
# (2 unchanged attributes hidden)
}
)
# (7 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
How to Reproduce
- Create the following schema
{
"namespace": "com.orga.NameSpace",
"name": "EnrichmentKey",
"type": "record",
"fields": [
{
"name": "code",
"type": "string"
}
]
}
and run terraform apply + plan.
- Add a new attribute to the previous schema and run terraform apply + plan:
{
"namespace": "com.orga.NameSpace",
"name": "EnrichmentKey",
"type": "record",
"fields": [
{
"name": "code",
"type": "string"
},
{
"name": "code2",
"type": "string"
}
]
}
- Delete attribute to the previous schema and terraform apply + plan:
{
"namespace": "com.orga.NameSpace",
"name": "EnrichmentKey",
"type": "record",
"fields": [
{
"name": "code",
"type": "string"
}
]
}
At this stage, apply succeeds, but plan returns a TF drift:
~ resource "confluent_schema" "key" {
id = "lsrc-3mndmj/enrichment-key-test-value/latest"
~ schema = jsonencode(
~ {
~ fields = [
{
name = "code"
type = "string"
},
- {
- name = "code2"
- type = "string"
},
]
name = "EnrichmentKey"
# (2 unchanged attributes hidden)
}
)
# (7 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Root Cause
It seems like the issue is TF uses loadIdForLatestSchema method:
curl --request GET \
--url 'https://....us-east-2.aws.confluent.cloud//subjects/enrichment-key-test-value/versions/latest' \
--header 'Authorization: Basic ...' | jq .
{
"subject": "enrichment-key-test-value",
"version": 2,
"id": 100014,
"schema": "{\"type\":\"record\",\"name\":\"EnrichmentKey\",\"namespace\":\"com.orga.NameSpace\",\"fields\":[{\"name\":\"code\",\"type\":\"string\"},{\"name\":\"code2\",\"type\":\"string\"}]}"
}
that would always return v2 and not v1 (v3). In order to fix it, we need to use version = -1 to force create a new version (3) to avoid:
curl --request GET \
--url 'https://....us-east-2.aws.confluent.cloud//subjects/enrichment-key-test-value/versions' \
--header 'Authorization: Basic ...' | jq .
[
1,
2
]
and make sure the first API call returns v3 instead of v2.