Transient errors during parallel plan from `.terraform.lock`: `plugins are not consistent`

[wyardley]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I'm occasionally getting some transient errors when running atlantis plan; currently, I have ATLANTIS_PARALLEL_POOL_SIZE set to 3. It's most typically on states that have a lot of providers, giving me two possible theories:

• all the states planning at once are causing the Terraform registry to rate-limit us, but the client is giving a confusing error message
• OR, it's a race condition due to parallel plans possibly using the same cache (Using integrations/foo vN.N.N from the shared cache directory in the output; see link below for why this is not guaranteed to be safe)

I'm not able to reproduce it right at the moment, and don't have the exact error handy, so I'll update here next time the issue comes up

Reproduction Steps

atlantis plan

Note: this is not consistently reproducible

Logs

n/a

Environment details

• Atlantis version: v0.19.7
• Atlantis flags: see below

Atlantis server-side config file:
All config is from env vars / flags (with some kube secret references / other irrelevant stuff omitted)

            - name: ATLANTIS_ATLANTIS_URL
              value: https://xxx/
            - name: ATLANTIS_DEFAULT_TF_VERSION
              value: "1.2.5"
            - name: ATLANTIS_HIDE_PREV_PLAN_COMMENTS
              value: "true"
            - name: ATLANTIS_PARALLEL_POOL_SIZE
              value: "3"
            - name: ATLANTIS_PORT
              value: "4141"
            - name: ATLANTIS_REPO_ALLOWLIST
              value: github.com/orgname/*
            - name: TF_CLI_ARGS_apply
              value: "-compact-warnings"
            - name: TF_CLI_ARGS_init
              value: "-compact-warnings"
            - name: TF_CLI_ARGS_plan
              value: "-compact-warnings"

Repo atlantis.yaml file:

---
version: 3
parallel_plan: true

projects:
# [...]

Any other information you can provide about the environment/deployment.
--->

Additional Context

• parallel init failure when using plugin cache hashicorp/terraform#25849

[wyardley]

Here's an example one.

Note:

• We do not check in our lockfiles, so any lockfile here is one that's local
• Re-running the plan typically resolves the issue

Warnings:

- Incomplete lock file information for providers

To see the full warning notes, run Terraform without -compact-warnings.

Terraform has been successfully initialized!
╷
│ Error: Required plugins are not installed
│ 
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/kubernetes: the cached package for registry.terraform.io/hashicorp/kubernetes 2.12.1 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
│ 
│ Terraform uses external plugins to integrate with a variety of different
│ infrastructure services. You must install the required plugins before
│ running Terraform operations.
╵

[jamengual]

is this still happening with v0.19.8?

[wyardley]

Hard to know since it's transient, but I would assume so. The more I think about it, the more I am convinced it's due to the plugin cache thing and the init happening in parallel.

I can think of some possible fixes, but they're beyond what I can implement. High level, though

• having a way to have init happen in serial, but plan in parallel (this might slow things down, but seems the simplest?)
• separate plugin cache dir per state
• some way of pre-downloading all the providers (e.g., using terraform providers mirror)

[wyardley]

Got this again just now:

Initializing the backend...

Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding gavinbunney/kubectl versions matching "1.14.0"...
- Finding integrations/github versions matching "4.29.0"...
- Finding hashicorp/google versions matching ">= 4.18.0, 4.33.0"...
- Finding hashicorp/kubernetes versions matching "~> 2.10, 2.13.0"...
- Finding fluxcd/flux versions matching "0.16.0"...
- Finding hashicorp/google-beta versions matching ">= 4.29.0, < 5.0.0"...
- Finding latest version of hashicorp/random...
- Using hashicorp/google-beta v4.33.0 from the shared cache directory
- Using hashicorp/random v3.3.2 from the shared cache directory
- Using gavinbunney/kubectl v1.14.0 from the shared cache directory
- Using integrations/github v4.29.0 from the shared cache directory
- Using hashicorp/google v4.33.0 from the shared cache directory
- Installing hashicorp/kubernetes v2.13.0...
- Installed hashicorp/kubernetes v2.13.0 (signed by HashiCorp)
- Using fluxcd/flux v0.16.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.


Warnings:

- Incomplete lock file information for providers

To see the full warning notes, run Terraform without -compact-warnings.

Terraform has been successfully initialized!
╷
│ Error: Required plugins are not installed
│ 
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/kubernetes: the cached package for registry.terraform.io/hashicorp/kubernetes 2.13.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
│ 
│ Terraform uses external plugins to integrate with a variety of different
│ infrastructure services. You must install the required plugins before
│ running Terraform operations.
╵

[nokernel]

@jamengual

is this still happening with v0.19.8?

I just got it with v0.19.8 and terraform 1.2.9.

What I saw in Atlantis terraform output is it did not fetch all the providers.

[vmdude]

We also have these Error: Required plugins are not installed issues, and it still happening in v0.22.2.

Is there a way to un-close this issue ?

[nitrocode]

The error shown by @wyardley

│ Error: Required plugins are not installed
│
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│ - registry.terraform.io/hashicorp/kubernetes: the cached package for registry.terraform.io/hashicorp/kubernetes 2.13.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file

Seems like that could be resolved by stomping over the .terraform.lock file prior to terraform init or doing terraform init -upgrade.

Is that the same issue you're hitting @vmdude ?

[vmdude]

.lock files are not versioned (through git I mean) on our side, and we're using the same cache dir for all parallel plans (same as issue creator).
Removing .lock file in the pre_workflow_hooks could not cause another race condition where we removing a lock file used by another parallel plan ?

[nitrocode]

I'm curious if you get the error if you run terraform init with the extra args -upgrade to stomp over the lock files on every run