CLI-3508: Persistent terraform drift when creating a confluent_schema that already existed as one of the previous versions

[linouk23]

What

There's a bug in the TF Provider that results in persistent Terraform drift when creating a confluent_schema that already existed as one of the previous versions. In other words, if a schema evolves into something that matches an old version of the schema, it won't get a new version. Instead, a user will experience persistent Terraform drift:

✗ terraform apply -auto-approve ...
...
Plan: 0 to add, 1 to change, 0 to destroy.
confluent_schema.key: Modifying... [id=lsrc-3mndmj/enrichment-key-test-value/latest]
confluent_schema.key: Still modifying... [id=lsrc-3mndmj/enrichment-key-test-value/latest, 10s elapsed]
confluent_schema.key: Modifications complete after 11s [id=lsrc-3mndmj/enrichment-key-test-value/latest]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

✗ terraform plan
...
  ~ update in-place
Terraform will perform the following actions:
  # confluent_schema.key will be updated in-place
  ~ resource "confluent_schema" "key" {
        id                          = "lsrc-3mndmj/enrichment-key-test-value/latest"
      ~ schema                      = jsonencode(
          ~ {
              ~ fields    = [
                    {
                        name = "code"
                        type = "string"
                    },
                  - {
                      - name = "code2"
                      - type = "string"
                    },
                ]
                name      = "EnrichmentKey"
                # (2 unchanged attributes hidden)
            }
        )
        # (7 unchanged attributes hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

How to Reproduce

1. Create the following schema

{
  "namespace": "com.orga.NameSpace",
  "name": "EnrichmentKey",
  "type": "record",
  "fields": [
    {
      "name": "code",
      "type": "string"
    }
  ]
}

and run terraform apply + plan.

2. Add a new attribute to the previous schema and run terraform apply + plan:

{
  "namespace": "com.orga.NameSpace",
  "name": "EnrichmentKey",
  "type": "record",
  "fields": [
    {
      "name": "code",
      "type": "string"
    },
    {
      "name": "code2",
      "type": "string"
    }
  ]
}

3. Delete attribute to the previous schema and terraform apply + plan:

{
  "namespace": "com.orga.NameSpace",
  "name": "EnrichmentKey",
  "type": "record",
  "fields": [
    {
      "name": "code",
      "type": "string"
    }
  ]
}

At this stage, apply succeeds, but plan returns a TF drift:

  ~ resource "confluent_schema" "key" {
        id                          = "lsrc-3mndmj/enrichment-key-test-value/latest"
      ~ schema                      = jsonencode(
          ~ {
              ~ fields    = [
                    {
                        name = "code"
                        type = "string"
                    },
                  - {
                      - name = "code2"
                      - type = "string"
                    },
                ]
                name      = "EnrichmentKey"
                # (2 unchanged attributes hidden)
            }
        )
        # (7 unchanged attributes hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Root Cause

It seems like the issue is TF uses loadIdForLatestSchema method:

curl --request GET \
  --url 'https://....us-east-2.aws.confluent.cloud//subjects/enrichment-key-test-value/versions/latest' \
  --header 'Authorization: Basic ...' | jq .
{
  "subject": "enrichment-key-test-value",
  "version": 2,
  "id": 100014,
  "schema": "{\"type\":\"record\",\"name\":\"EnrichmentKey\",\"namespace\":\"com.orga.NameSpace\",\"fields\":[{\"name\":\"code\",\"type\":\"string\"},{\"name\":\"code2\",\"type\":\"string\"}]}"
}

that would always return v2 and not v1 (v3). In order to fix it, we need to use version = -1 to force create a new version (3) to avoid:

curl --request GET \
  --url 'https://....us-east-2.aws.confluent.cloud//subjects/enrichment-key-test-value/versions' \
  --header 'Authorization: Basic ...' | jq .
[
  1,
  2
]

and make sure the first API call returns v3 instead of v2.

[linouk23]

Update: we ran into an issue with #620 as setting version=-1 updates the metadata.properties attribute for a schema as well which makes it challenging to avoid TF drift as TF config has got metadata.properties = {} initially:

{
  "subject": "enrichment-key-test-value",
  "version": 3,
  "id": 100015,
  "metadata": {
    "properties": {
      "confluent:version": "3"
    }
  },
  "schema": "{\"type\":\"record\",\"name\":\"EnrichmentKey\",\"namespace\":\"com.orga.NameSpace\",\"fields\":[{\"name\":\"code\",\"type\":\"string\"}]}"
}

Instead, we recommend the following as a short-term workaround:

Workaround №1 (for JSON & AVRO schemas):

1. Insert "description": "Sample schema to help you get started." (JSON) or "doc": "Sample schema to help you get started." (AVRO) directly into your JSON / AVRO file to initiate the creation of a new schema version. If this attribute already exists, simulate an edit by adding an extra whitespace or a period to it. This method allows you to edit the JSON / AVRO file, which is beneficial because it avoids editing attributes of an instance of the confluent_schema resource. This is particularly useful if you're using Terraform modules or for_each constructs that reuse the same definition of the confluent_schema resource.

AVRO example:

{
  "type": "record",
  "namespace": "com.mycorp.mynamespace",
  "name": "sampleRecord",
  "doc": "Sample schema to help you get started.",
  "fields": [
    {
      "name": "my_field1",
      "type": "int",
      "doc": "The int type is a 32-bit signed integer."
    }
  ]
}

JSON example:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://example.com/myURI.schema.json",
  "title": "SampleRecord",
  "description": "Sample schema to help you get started.",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "myField1": {
      "type": "integer",
      "description": "The integer type is used for integral numbers."
    }
  }
}

Workaround №2 (alternative option):

1. If you can see this TF drift, you should open CC UI and find the current version for your subject.
   For example, if you use this screenshot, version = 2:

You can also use the confluent_schema data source to find the version number, or you can find it in the TF state by running the command terraform state show confluent_schema.main.example.

2. Now, set the metadata.properties attribute in your TF config and explicitly increment the version:

# ./schemas/avro/key.avsc refers to the updated schema (v3)
resource "confluent_schema" "key" {
  subject_name = "enrichment-key-issue-123-value"
  format       = "AVRO"
  schema       = file("./schemas/avro/key.avsc")

  metadata {
    properties = {
      "confluent:version" : "3"
    }
  }
}

Here's what you'd see after applying this change:

3. If you do decide to update to schema v4 that doesn't exist in the previous version, feel free to update both ./schemas/avro/key.avsc and remove metadata.properties block completely, and then run terraform plan and terraform apply:

# ./schemas/avro/key.avsc refers to the updated schema (v4)
resource "confluent_schema" "key" {
  subject_name = "enrichment-key-issue-123-value"
  format       = "AVRO"
  schema       = file("./schemas/avro/key.avsc")
}

Note 1: We did test this workaround, and the results look promising.

Note 2: Essentially, this is pretty much what version = -1 accomplishes, but it is more compatible with the current implementation of the TF Provider, as it maintains consistency between the TF config and API response. Otherwise, there's this issue of "drifting" within the metadata.properties, which would exist in the response but not in the TF config if the TF Provider hardcodes version = -1.