[Terraform Credentials] Expirations of multiple credentials (backends, API) in jenkins-infra/terraform-states

[lemeurherve]

Previous rotation: #4895

A bunch of credentials expires on 08 march 2026 (yes, yesterday :'() in the (private) repository jenkins-infra/terraform-states.

Tip

This private repository manages (using Terraform code) the Terraform State Backend (today using Azure File Share) and project's API tokens (Azure API token for terraform CLI, Cloudflare, DigitalOcean, etc.).

---

For each credential, we have to:

• Identify the controller and job(s) using the credential
• Merge and then apply (manually) the (private) PR which renew the secrets (with a new 3 month expiration date)
• Update the secret in the Kubernetes Management Chart Secrets for the corresponding Jenkins controller which has the jobs requiring these credentials
  • ⚠️ There is a tedious update process due to how Kubernetes and JCasC work: Sops file update pushed -> kubernetes-management run to change Kubernetes Secret's value -> kubelet updates the symlink -> JCasC reload (triggered by sidecar kiwigrid and/or human) to load in the controller JVM memory -> MB Job scan to load the new value from controller JVM memory to job configuration -> run the job with new value
• Ensure the job, using the credential, can be run successfully with the new credential

---

Task list:

• AWS (Sponsored): https://github.com/jenkins-infra/terraform-states/pull/85
• Azure: https://github.com/jenkins-infra/terraform-states/pull/83
• Azure-net: https://github.com/jenkins-infra/terraform-states/pull/84
• Cloudflare: https://github.com/jenkins-infra/terraform-states/pull/86
• Datadog: https://github.com/jenkins-infra/terraform-states/pull/89
• DigitalOcean: https://github.com/jenkins-infra/terraform-states/pull/87
• Fastly: https://github.com/jenkins-infra/terraform-states/pull/88
• Create calendar event

[lemeurherve]

Asked for help in #jenkins-infra to apply azure-net change as I don't have enough permissions to do so.

[lemeurherve]

For azure, the plan is showing me more than expected from the credentials renewal, I haven't applied it yet:

Plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # azapi_update_resource.encryptionathost will be created
  + resource "azapi_update_resource" "encryptionathost" {
      + body                    = {
          + properties = {}
        }
      + id                      = (known after apply)
      + ignore_casing           = false
      + ignore_missing_property = true
      + name                    = (known after apply)
      + output                  = (known after apply)
      + parent_id               = (known after apply)
      + resource_id             = "/subscriptions/1e7d5219-acbc-4495-8629-bdbb22e9b3ed/providers/Microsoft.Features/featureProviders/Microsoft.Compute/subscriptionFeatureRegistrations/encryptionathost"
      + sensitive_body          = (write-only attribute)
      + type                    = "Microsoft.Features/featureProviders/subscriptionFeatureRegistrations@2021-07-01"
    }

  # azuread_application_password.terraform_app_password["production"] must be replaced
-/+ resource "azuread_application_password" "terraform_app_password" {
      ~ end_date       = "2026-03-08T00:00:00Z" -> "2026-06-07T00:00:00Z" # forces replacement
      ~ id             = "e651c485-66cf-4f54-aaa4-8d2c895a6bc3/password/ba51a227-1772-43f1-b993-cec484e11b4c" -> (known after apply)
      ~ key_id         = "ba51a227-1772-43f1-b993-cec484e11b4c" -> (known after apply)
      ~ start_date     = "2025-12-08T09:23:54.8925811Z" -> (known after apply)
      ~ value          = (sensitive value)
        # (2 unchanged attributes hidden)
    }

  # azuread_application_password.terraform_app_password["staging"] must be replaced
-/+ resource "azuread_application_password" "terraform_app_password" {
      ~ end_date       = "2026-03-08T00:00:00Z" -> "2026-06-07T00:00:00Z" # forces replacement
      ~ id             = "f24ebfa8-c774-41ea-ba66-254ce06e71cd/password/61a54ced-b9dc-481d-b54b-39864637f042" -> (known after apply)
      ~ key_id         = "61a54ced-b9dc-481d-b54b-39864637f042" -> (known after apply)
      ~ start_date     = "2025-12-08T09:23:52.2032471Z" -> (known after apply)
      ~ value          = (sensitive value)
        # (2 unchanged attributes hidden)
    }

  # azurerm_role_assignment.terraform_role_assignement_contributor_azurerm["production"] will be created
  + resource "azurerm_role_assignment" "terraform_role_assignement_contributor_azurerm" {
      + condition_version                = (known after apply)
      + id                               = (known after apply)
      + name                             = (known after apply)
      + principal_id                     = "b847a030-25e1-4791-ad04-9e8484d87bce"
      + principal_type                   = (known after apply)
      + role_definition_id               = (known after apply)
      + role_definition_name             = "Owner"
      + scope                            = "/subscriptions/dff2ec18-6a8e-405c-8e45-b7df7465acf0"
      + skip_service_principal_aad_check = (known after apply)
    }

  # azurerm_role_assignment.terraform_role_assignement_contributor_azurerm["staging"] will be created
  + resource "azurerm_role_assignment" "terraform_role_assignement_contributor_azurerm" {
      + condition_version                = (known after apply)
      + id                               = (known after apply)
      + name                             = (known after apply)
      + principal_id                     = "0cf969a1-836c-45c3-9290-f17a675cfc71"
      + principal_type                   = (known after apply)
      + role_definition_id               = (known after apply)
      + role_definition_name             = "Owner"
      + scope                            = "/subscriptions/dff2ec18-6a8e-405c-8e45-b7df7465acf0"
      + skip_service_principal_aad_check = (known after apply)
    }

  # azurerm_role_assignment.terraform_role_assignement_contributor_azurerm_jenkins_sponsored["production"] will be created
  + resource "azurerm_role_assignment" "terraform_role_assignement_contributor_azurerm_jenkins_sponsored" {
      + condition_version                = (known after apply)
      + id                               = (known after apply)
      + name                             = (known after apply)
      + principal_id                     = "b847a030-25e1-4791-ad04-9e8484d87bce"
      + principal_type                   = (known after apply)
      + role_definition_id               = (known after apply)
      + role_definition_name             = "Owner"
      + scope                            = "/subscriptions/1e7d5219-acbc-4495-8629-bdbb22e9b3ed"
      + skip_service_principal_aad_check = (known after apply)
    }

  # azurerm_role_assignment.terraform_role_assignement_contributor_azurerm_jenkins_sponsored["staging"] will be created
  + resource "azurerm_role_assignment" "terraform_role_assignement_contributor_azurerm_jenkins_sponsored" {
      + condition_version                = (known after apply)
      + id                               = (known after apply)
      + name                             = (known after apply)
      + principal_id                     = "0cf969a1-836c-45c3-9290-f17a675cfc71"
      + principal_type                   = (known after apply)
      + role_definition_id               = (known after apply)
      + role_definition_name             = "Owner"
      + scope                            = "/subscriptions/1e7d5219-acbc-4495-8629-bdbb22e9b3ed"
      + skip_service_principal_aad_check = (known after apply)
    }

  # module.backends["production"].azuread_application_password.app_password must be replaced
-/+ resource "azuread_application_password" "app_password" {
      ~ end_date       = "2026-03-08T00:00:00Z" -> "2026-06-07T00:00:00Z" # forces replacement
      ~ id             = "90b28efc-ee97-4e98-9357-f28cf80d9480/password/a743b3aa-38d5-4496-8ced-fe45de6d5c22" -> (known after apply)
      ~ key_id         = "a743b3aa-38d5-4496-8ced-fe45de6d5c22" -> (known after apply)
      ~ start_date     = "2025-12-08T09:23:52.9426224Z" -> (known after apply)
      ~ value          = (sensitive value)
        # (2 unchanged attributes hidden)
    }

  # module.backends["staging"].azuread_application_password.app_password must be replaced
-/+ resource "azuread_application_password" "app_password" {
      ~ end_date       = "2026-03-08T00:00:00Z" -> "2026-06-07T00:00:00Z" # forces replacement
      ~ id             = "6ee443e8-8621-48ea-a753-38645f31a2e6/password/883a2f40-8683-4899-a1d8-95de920ba6a1" -> (known after apply)
      ~ key_id         = "883a2f40-8683-4899-a1d8-95de920ba6a1" -> (known after apply)
      ~ start_date     = "2025-12-08T09:23:53.3647665Z" -> (known after apply)
      ~ value          = (sensitive value)
        # (2 unchanged attributes hidden)
    }

Plan: 9 to add, 0 to change, 4 to destroy.

It was due to a leftover terraform.state, deleting it fixed that.

[lemeurherve]

aws-sponsored

• Merged https://github.com/jenkins-infra/terraform-states/pull/85
• Retrieved last terraform-states commit in my local repo
• Ran make terraform-apply TERRAFORM_DIR=./aws-sponsored/
• Pushed¹ https://github.com/jenkins-infra/charts-secrets/commit/05141572ebaf4460b440abd85f0cf196188be8b0
• Pushed https://github.com/jenkins-infra/terraform-states/commit/7ff4db27c21549e02704979da173bb83b19b38b0
• Ran kubernetes-management main job and reloaded jcasc config
• Got the following error when trying to run terraform aws-sponsored main job:

Storage Account Name: "awssponsoredprodu4bcywi"): authorizing request: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '2a3a8aee-90af-49ab-b71b-5b1b634247de'. Trace ID: 67e65820-c311-4ac4-9c22-8a771c412900 Correlation ID: a7729775-ae0d-422a-bf4c-fbd8c4c20b39 Timestamp: 2026-03-09 10:49:46Z","error_codes":[7000215],"timestamp":"2026-03-09 10:49:46Z","trace_id":"67e65820-c311-4ac4-9c22-8a771c412900","correlation_id":"a7729775-ae0d-422a-bf4c-fbd8c4c20b39","error_uri":"https://login.microsoftonline.com/error?code=7000215"}

EDIT: after azure renewal, all good now ✅

Just got some unrelated changes in the plan:

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.cijenkinsio_agents_2.aws_eks_cluster.this[0] has changed
  ~ resource "aws_eks_cluster" "this" {
        id                            = "cijenkinsio-agents-2"
        name                          = "cijenkinsio-agents-2"
      ~ platform_version              = "eks.24" -> "eks.27"
        tags                          = {
            "GithubOrg"          = "jenkins-infra"
            "GithubRepo"         = "terraform-aws-sponsorship"
            "associated_service" = "eks/cijenkinsio-agents-2"
            "repository"         = "jenkins-infra/terraform-aws-sponsorship"
            "scope"              = "terraform-managed"
        }
        # (13 unchanged attributes hidden)

        # (6 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.cijenkinsio_agents_2.module.eks_managed_node_group["applications"].aws_eks_node_group.this[0] will be updated in-place
  ~ resource "aws_eks_node_group" "this" {
        id                     = "cijenkinsio-agents-2:applications-20250627071357253400000018"
      ~ release_version        = "1.33.5-20260129" -> "1.33.8-20260304"
        tags                   = {
            "GithubOrg"          = "jenkins-infra"
            "GithubRepo"         = "terraform-aws-sponsorship"
            "Name"               = "applications"
            "associated_service" = "eks/cijenkinsio-agents-2"
            "repository"         = "jenkins-infra/terraform-aws-sponsorship"
            "scope"              = "terraform-managed"
        }
        # (16 unchanged attributes hidden)

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Footnotes

1. I pushed the charts-secrets commit before the terraform-states one by mistake. ↩

[lemeurherve]

azure-net and azure OK ✅

(with the help of @dduportal 🍯)