AzureRM backend OIDC authentication hangs during terraform init stage within Github workflow

[delikvent]

Terraform Version

Terraform v1.5.6

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.71.0"
    }
  }
  backend "azurerm" {}
}

provider "azurerm" {
  features {}
}

data "azurerm_resource_group" "rg" {
  name = "rg-example-1"
}

Debug Output

https://gist.github.com/delikvent/caf426f7c34553adce018b25b314c6c8

Expected Behavior

terraform init stage within github workflow should fail if no matching federated identity record is found for presented assertion.

Actual Behavior

terraform init stage within github workflow hangs (seemingly forever).

Steps to Reproduce

terraform init
-backend-config=subscription_id=REDACTED
-backend-config=resource_group_name=REDACTED
-backend-config=storage_account_name=REDACTED
-backend-config=container_name=REDACTED
-backend-config=key=REDACTED
-input=false

Additional Context

shell:
bash

environment_variables:
ARM_USE_OIDC=TRUE
ARM_CLIENT_ID=REDACTED
ARM_TENANT_ID=REDACTED

github_workflow_permissions:
id-token: write
contents: read

additional_info:
When performing authentication via "az login" instead of "terraform" core binary we can see the expected error message:
<<- ERROR MESSAGE START ->>
AADSTS70021: No matching federated identity record found for presented assertion.
Assertion Issuer: "https://token.actions.githubusercontent.com"
Assertion Subject: REDACTED
Assertion Audience: "api://AzureADTokenExchange"
<<- ERROR MESSAGE END ->>

References

No response

[crw]

Thanks for this bug report!

If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!

[kgibson-insight]

I'm having the same issue on v1.6.3

here is trace

2023-11-02T20:01:53.475Z [INFO]  Terraform version: 1.6.3
2023-11-02T20:01:53.475Z [DEBUG] using github.com/hashicorp/go-tfe v1.36.0
2023-11-02T20:01:53.475Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.19.1
2023-11-02T20:01:53.475Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2023-11-02T20:01:53.475Z [DEBUG] using github.com/zclconf/go-cty v1.14.1
2023-11-02T20:01:53.475Z [INFO]  Go runtime version: go1.21.3
2023-11-02T20:01:53.475Z [INFO]  CLI args: []string{"terraform", "init", "-backend-config=backend.tfvars"}
2023-11-02T20:01:53.475Z [TRACE] Stdout is a terminal of width 188
2023-11-02T20:01:53.475Z [TRACE] Stderr is a terminal of width 188
2023-11-02T20:01:53.475Z [TRACE] Stdin is a terminal
2023-11-02T20:01:53.475Z [DEBUG] Attempting to open CLI config file: /home/vsts/.terraformrc
2023-11-02T20:01:53.475Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2023-11-02T20:01:53.475Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2023-11-02T20:01:53.475Z [DEBUG] ignoring non-existing provider search directory /home/vsts/.terraform.d/plugins
2023-11-02T20:01:53.475Z [DEBUG] ignoring non-existing provider search directory /home/vsts/.local/share/terraform/plugins
2023-11-02T20:01:53.475Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2023-11-02T20:01:53.476Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2023-11-02T20:01:53.476Z [DEBUG] ignoring non-existing provider search directory /var/lib/snapd/desktop/terraform/plugins
2023-11-02T20:01:53.476Z [INFO]  CLI command args: []string{"init", "-backend-config=backend.tfvars"}

Initializing the backend...
2023-11-02T20:01:53.480Z [TRACE] Meta.Backend: merging -backend-config=... CLI overrides into backend configuration
2023-11-02T20:01:53.480Z [TRACE] Meta.Backend: built configuration for "azurerm" backend with hash value 3206449086
2023-11-02T20:01:53.480Z [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2023-11-02T20:01:53.480Z [DEBUG] New state was assigned lineage "5a4553ea-f389-4894-9f87-e7e7cbc99335"
2023-11-02T20:01:53.480Z [TRACE] Meta.Backend: moving from default local state only to "azurerm" backend
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-11-02T20:01:53.482Z [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-11-02T20:01:53.482Z [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-11-02T20:01:53.482Z [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-11-02T20:01:53.482Z [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2023-11-02T20:01:53.482Z [TRACE] getproviders.SearchLocalDirectory: failed to resolve symlinks for .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.482Z [TRACE] providercache.fillMetaCache: error while scanning directory .terraform/providers: cannot search .terraform/providers: lstat .terraform/providers: no such file or directory
2023-11-02T20:01:53.483Z [DEBUG] checking for provisioner in "."
2023-11-02T20:01:53.483Z [DEBUG] checking for provisioner in "/opt/hostedtoolcache/terraform/1.6.3/x64"
2023-11-02T20:01:53.483Z [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2023-11-02T20:01:53.483Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2023-11-02T20:01:53.483Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2023-11-02T20:01:53.483Z [TRACE] statemgr.Filesystem: read nil snapshot
2023-11-02T20:01:53.483Z [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2023-11-02T20:01:53.484Z [INFO]  Testing if Service Principal / Client Certificate is applicable for Authentication..
2023-11-02T20:01:53.484Z [INFO]  Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..
2023-11-02T20:01:53.484Z [INFO]  Testing if Service Principal / Client Secret is applicable for Authentication..
2023-11-02T20:01:53.484Z [INFO]  Testing if OIDC is applicable for Authentication..
2023-11-02T20:01:53.484Z [INFO]  Using OIDC for Authentication
2023-11-02T20:01:53.484Z [INFO]  Getting OAuth config for endpoint https://login.microsoftonline.com/ with  tenant 5a4553ea-f389-4894-9f87-e7e7cbc99335
2023-11-02T20:01:53.484Z [DEBUG] Obtaining an MSAL / Microsoft Graph token for Resource Manager..
2023-11-02T20:01:53.484Z [DEBUG] New state was assigned lineage "5a4553ea-f389-4894-9f87-e7e7cbc99335"
2023-11-02T20:01:53.484Z [DEBUG] Building the Container Client from an Access Token (using user credentials)

[ybensoussan]

Found this issue, the setup is a bit finicky. What worked for me is the following:

in the backend config you need:

subscription_id              =  "00000-0000-0000-0000-00000000" # subscription with state storage account
resource_group_name   = "state_rg_name"
storage_account_name = "state_container"
container_name            = "tfstate"
key                                = "my_awesome_state.tfstate"
client_id                        = "00000-0000-0000-0000-00000000" # client_id that is setup with oidc and privileges to the state
use_oidc                       = true
tenant_id                      = "00000-0000-0000-0000-00000000" 

Don't set ANY ARM_** environment variables (The only one I have is ARM_TENANT_ID, but this can also be included in the backend file as above.

With this the backend seems to initialize fine trough OIDC now.
And subsequent provider blocks in the plan can be used with different client_id sourced from variables or ENV variables.

Which looks like this in GH actions and the provider config:

# Github actions config

      - name: Terraform Plan
        if: inputs.tf_action == 'plan'
        id: plan
        run: terraform plan -var-file=<(cat ../vars/${{ env.tenant }}/*.tfvars) -out=tfplan
        working-directory: ${{ env.plan }}/plan
        env:
          ARM_TENANT_ID: ${{ vars.ARM_TENANT_ID }}
          TF_VAR_client_id_1: ${{ vars.ID1 }}
          TF_VAR_client_id_2: ${{ vars.ID2 }}
          TF_VAR_client_id_3: ${{ vars.ID3 }}


# Provider config          
provider "azurerm" {
  alias           = "ID1"
  use_oidc        = true
  subscription_id = var.target_subscription_id
  client_id       = var.client_id_1
  features {}
  skip_provider_registration = true
}

provider "azurerm" {
  alias           = "ID2"
  use_oidc        = true
  subscription_id = var.target_subscription_id
  client_id       = var.client_id_2
  features {}
  skip_provider_registration = true
}

provider "azuread" {
  alias     = "ID3"
  use_oidc  = true
  client_id = var.client_id_3
}