`terraform init` does not recognize the most recent version of a registry.terraform.io hosted module

[jwadolowski]

Terraform Version

Terraform v1.11.1
on linux_arm64

Terraform Configuration Files

N/A

Debug Output

$ docker run -it -w /workspace -v ${PWD}/main.tf:/workspace/main.tf -e TF_LOG=DEBUG hashicorp/terraform:1.11.1 init
2025-03-11T14:48:59.651Z [INFO]  Terraform version: 1.11.1
2025-03-11T14:48:59.651Z [DEBUG] using github.com/hashicorp/go-tfe v1.70.0
2025-03-11T14:48:59.651Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.23.0
2025-03-11T14:48:59.651Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.1.1
2025-03-11T14:48:59.651Z [DEBUG] using github.com/zclconf/go-cty v1.16.0
2025-03-11T14:48:59.651Z [INFO]  Go runtime version: go1.23.3
2025-03-11T14:48:59.651Z [INFO]  CLI args: []string{"/bin/terraform", "init"}
2025-03-11T14:48:59.651Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2025-03-11T14:48:59.651Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2025-03-11T14:48:59.651Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2025-03-11T14:48:59.651Z [DEBUG] ignoring non-existing provider search directory /root/.terraform.d/plugins
2025-03-11T14:48:59.651Z [DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
2025-03-11T14:48:59.651Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2025-03-11T14:48:59.651Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2025-03-11T14:48:59.651Z [INFO]  CLI command args: []string{"init"}
Initializing the backend...
2025-03-11T14:48:59.652Z [DEBUG] checking for provisioner in "."
2025-03-11T14:48:59.653Z [DEBUG] checking for provisioner in "/bin"
Initializing modules...
2025-03-11T14:48:59.654Z [DEBUG] Module installer: begin cloudfront-s3-cdn
2025-03-11T14:48:59.654Z [DEBUG] cloudfront-s3-cdn listing available versions of registry.terraform.io/cloudposse/cloudfront-s3-cdn/aws at registry.terraform.io
2025-03-11T14:48:59.654Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2025-03-11T14:48:59.698Z [DEBUG] fetching module versions from "https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions"
2025-03-11T14:48:59.698Z [DEBUG] GET https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.1.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.1.1" for cloudposse/cloudfront-s3-cdn/aws
...
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.93.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.93.1" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.94.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.95.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.95.1" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.96.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.96.1" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:48:59.830Z [DEBUG] found available version "0.96.2" for cloudposse/cloudfront-s3-cdn/aws
╷
│ Error: Unresolvable module version constraint
│
│   on main.tf line 1:
│    1: module "cloudfront-s3-cdn" {
│
│ There is no available version of module "registry.terraform.io/cloudposse/cloudfront-s3-cdn/aws" (main.tf:1) which matches the given version constraint. The newest available version is
│ 0.96.2.
╵

Expected Behavior

At the time of writing, 0.97.0 is the most recent version of the cloudposse/cloudfront-s3-cdn/aws module, therefore it should be downloaded.

Actual Behavior

Terraform 1.11.1 claims the latest available version of the cloudposse/cloudfront-s3-cdn/aws module is 0.96.2, while both 1.11.0 and 1.10.5 correctly recognize 0.97.0 as the most recent one.

Steps to Reproduce

1. Create main.tf

module "cloudfront-s3-cdn" {
  source  = "cloudposse/cloudfront-s3-cdn/aws"
  version = "0.97.0"
}

2. Use Terrafrom 1.11.1 to init the working directory

$ docker run -it -w /workspace -v ${PWD}/main.tf:/workspace/main.tf -e TF_LOG=DEBUG hashicorp/terraform:1.11.1 init
2025-03-11T14:55:25.426Z [INFO]  Terraform version: 1.11.1
...
2025-03-11T14:55:25.592Z [DEBUG] found available version "0.96.0" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:55:25.592Z [DEBUG] found available version "0.96.1" for cloudposse/cloudfront-s3-cdn/aws
2025-03-11T14:55:25.592Z [DEBUG] found available version "0.96.2" for cloudposse/cloudfront-s3-cdn/aws
╷
│ Error: Unresolvable module version constraint
│
│   on main.tf line 1:
│    1: module "cloudfront-s3-cdn" {
│
│ There is no available version of module "registry.terraform.io/cloudposse/cloudfront-s3-cdn/aws" (main.tf:1) which matches the given version constraint. The newest available version is
│ 0.96.2.
╵

3. Do the same using 1.11.0

$ docker run -it -w /workspace -v ${PWD}/main.tf:/workspace/main.tf hashicorp/terraform:1.11.0 init
Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/cloudposse/cloudfront-s3-cdn/aws 0.97.0 for cloudfront-s3-cdn...
- cloudfront-s3-cdn in .terraform/modules/cloudfront-s3-cdn
...
Downloading registry.terraform.io/cloudposse/label/null 0.25.0 for cloudfront-s3-cdn.this...
- cloudfront-s3-cdn.this in .terraform/modules/cloudfront-s3-cdn.this
Initializing provider plugins...
- Finding hashicorp/random versions matching ">= 2.2.0"...
- Finding hashicorp/time versions matching ">= 0.7.0"...
- Finding hashicorp/aws versions matching ">= 2.0.0, >= 4.9.0"...
- Finding hashicorp/local versions matching ">= 1.2.0"...
- Installing hashicorp/random v3.7.1...
- Installed hashicorp/random v3.7.1 (signed by HashiCorp)
- Installing hashicorp/time v0.13.0...
- Installed hashicorp/time v0.13.0 (signed by HashiCorp)
- Installing hashicorp/aws v5.90.1...
- Installed hashicorp/aws v5.90.1 (signed by HashiCorp)
- Installing hashicorp/local v2.5.2...
- Installed hashicorp/local v2.5.2 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

5. And one more time with 1.10.5

$ docker run -it -w /workspace -v ${PWD}/main.tf:/workspace/main.tf hashicorp/terraform:1.10.5 init
Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/cloudposse/cloudfront-s3-cdn/aws 0.97.0 for cloudfront-s3-cdn...
- cloudfront-s3-cdn in .terraform/modules/cloudfront-s3-cdn
...
Downloading registry.terraform.io/cloudposse/label/null 0.25.0 for cloudfront-s3-cdn.this...
- cloudfront-s3-cdn.this in .terraform/modules/cloudfront-s3-cdn.this
Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 2.0.0, >= 4.9.0"...
- Finding hashicorp/local versions matching ">= 1.2.0"...
- Finding hashicorp/random versions matching ">= 2.2.0"...
- Finding hashicorp/time versions matching ">= 0.7.0"...
- Installing hashicorp/aws v5.90.1...
- Installed hashicorp/aws v5.90.1 (signed by HashiCorp)
- Installing hashicorp/local v2.5.2...
- Installed hashicorp/local v2.5.2 (signed by HashiCorp)
- Installing hashicorp/random v3.7.1...
- Installed hashicorp/random v3.7.1 (signed by HashiCorp)
- Installing hashicorp/time v0.13.0...
- Installed hashicorp/time v0.13.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Additional Context

1. It seems to be a regression issue introduced in 1.11.1
2. Version 0.97.0 of the module was released yesterday: https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/releases/tag/v0.97.0
3. The 0.97.0 does show up in the API response

$ curl -s 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' | jq '.modules[0].versions[].version' | tail
"0.92.1"
"0.93.0"
"0.93.1"
"0.94.0"
"0.95.0"
"0.95.1"
"0.96.0"
"0.96.1"
"0.96.2"
"0.97.0"

4. 0.97.0 (the latest) and 0.96.2 (previous version) look nearly identical from API perspective (all the required fields are present):

# 0.96.2
curl -s 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' | jq '.modules[0].versions[] | select(.version == "0.96.2")'

# 0.97.0
curl -s 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' | jq '.modules[0].versions[] | select(.version == "0.97.0")'

References

No response

Generative AI / LLM assisted development?

N/A

[jbardin]

Hi @jwadolowski,

Thanks for filing the issue! I have a feeling this is somehow related to the CDN cache, because I don't see any reason the particular Terraform version is involved. Terraform v1.11.1 and main work from the CLI, and curl can fetch the versions from the cli and within the distributed docker image. It will take some debugging to see why the combination of Terraform version and docker image isn't seeing 0.97.0, but we also should have a new release soon which may be able to give us another data point.

[jwadolowski]

Hi @jbardin,

CDN cache glitch was my very first thought as well, but it was surprisingly consistent, no matter what location I connected from (it may or may not matter in this case, it all depends how your CDN was configured).

I tried 1.11.1 with quite a few different locations and the result was always the same no matter what CloudFront POP handled my requests. Here are a few city / CloudFront POP combinations I routed my requests through:

• London / MAN50-C3
• Frankfurt / DUS51-P1
• New York / PHL51-P1
• Sydney / SYD62-P1
• and so on

There was a common denominator for all of them:

2025-03-11 20:25:35 GET https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions HTTP/2.0
                        ← 200 application/json 2.5k 503ms
content-type:               application/json
content-encoding:           gzip
last-modified:              Mon, 24 Feb 2025 20:00:46 GMT
server:                     terraform-registry/0810e3a503927d243774f0798994672c03f955d9
strict-transport-security:  max-age=31536000; includeSubDomains; preload
date:                       Mon, 10 Mar 2025 10:11:10 GMT
cache-control:              public, max-age=30, stale-while-revalidate=1800, stale-if-error=31536000
vary:                       Accept-Encoding
x-cache:                    RefreshHit from cloudfront
via:                        1.1 bac8af6ab43417aff0768ef23a8c05de.cloudfront.net (CloudFront)
x-amz-cf-pop:               SYD62-P1
x-amz-cf-id:                zGM38ZZkT4sY8bnMazY8kWajRIumNzbQ6DFDy3dUZ1daFQW0jcPe4g==
age:                        119665

The eye-striking thing is the Age header which is always greater than 24 hours.

As far as I can see the request initiated by terraform CLI to https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions always includes the following headers:

• user-agent: Terraform/<VERSION>
• x-terraform-version: <VERSION>
• accept-encoding: gzip

Here's curl's equivalent of terraform initiated request:

$ curl 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' --compressed -H 'x-terraform-version: 1.11.1' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version') -A "Terraform/1.11.1"
HTTP/2 200
content-type: application/json
content-encoding: gzip
last-modified: Mon, 24 Feb 2025 20:00:46 GMT
server: terraform-registry/0810e3a503927d243774f0798994672c03f955d9
strict-transport-security: max-age=31536000; includeSubDomains; preload
date: Mon, 10 Mar 2025 10:11:10 GMT
cache-control: public, max-age=30, stale-while-revalidate=1800, stale-if-error=31536000
vary: Accept-Encoding
x-cache: RefreshHit from cloudfront
via: 1.1 56f08e51c16f365de3e0991809e86e7c.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P5
x-amz-cf-id: yauAxQSESt9Olr12bvOVv9MnQ8UUtNhqHjnMGv05OxFIKPvbJ4Sgwg==
age: 120588

jq did not find 0.97.0 in the JSON returned by the API (so far it's consistent), however as soon as I started modifying the request the most recent version started to show up:

# --------------
# Let's assume that Terraform's HTTP client does not support HTTP compression 
# --------------
$ curl 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' -H 'x-terraform-version: 1.11.1' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version') -A "Terraform/1.11.1"
HTTP/2 200
content-type: application/json
last-modified: Mon, 10 Mar 2025 18:34:53 GMT
server: terraform-registry/5bfe2004d7775caf1927b13288e2409add4ca4ec
strict-transport-security: max-age=31536000; includeSubDomains; preload
date: Tue, 11 Mar 2025 19:42:20 GMT
cache-control: public, max-age=30, stale-while-revalidate=1800, stale-if-error=31536000
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 5a79618911a270a80c56d093cac91944.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P5
x-amz-cf-id: 3z45c7wbIpA-mPoCgI-LGJ8WlU09ZwLjvbGOLa0CF6cbKse_pb3U8w==
age: 125

"0.97.0"

# --------------
# A request that Terraform 1.11.0 would send
# --------------
$ curl 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' --compressed -H 'x-terraform-version: 1.11.0' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version') -A "Terraform/1.11.0"
HTTP/2 200
content-type: application/json
content-encoding: gzip
last-modified: Mon, 10 Mar 2025 18:34:53 GMT
server: terraform-registry/5bfe2004d7775caf1927b13288e2409add4ca4ec
strict-transport-security: max-age=31536000; includeSubDomains; preload
date: Tue, 11 Mar 2025 20:08:17 GMT
cache-control: public, max-age=30, stale-while-revalidate=1800, stale-if-error=31536000
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 95ff0d830848b741160e24f658d880e8.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS58-P1
x-amz-cf-id: bCZV-oc1fFWTzkYJP8HvRRlZgwCAYsuYvTlMT7egXeVDrcaE_HpbNA==
age: 1336

"0.97.0"

In comparison to the original request, the Age header value is much smaller and matches to what cache-control suggests - cache for 30s by default, allow stale data up to 30m and serve stale for 1y in case of upstream server errors.

All in all, it seems that your CDN configuration takes both Accept-Encoding (not a surprise at all, it's a fairy standard thing) and x-terraform-version (custom configuration) into consideration when calculating CDN cache keys.

For some reason, the CDN object that maps to key(host, path, "Accept-Encoding: gzip", "x-terraform-version: 1.11.1") got cached for much longer than anticipated. I can only guess what could be the root cause here, but server header may indicate some backend changes (or a deployment?):

• the stale object (without 0.97.0 version in the output) always has terraform-registry/0810e3a503927d243774f0798994672c03f955d9 (regardless of the HTTP client location)
• correct (up-to-date) object was served by terraform-registry/5bfe2004d7775caf1927b13288e2409add4ca4ec

[jwadolowski]

The bottom line is that most likely, the new version of Terraform will fix the issue (new x-terraform-version header value implies new cache key). However, it would be great to figure out why the response got cached for so long and what can be done to prevent similar issues in the future. The Last-Modified header says the stale object was changed on Mon, 24 Feb 2025 20:00:46 GMT - does this correlate to any event on HashiCorp's side (registry deployment, configuration change, etc.)?

Since User-Agent is not included in the cache key, either the x-terraform-version header or the URL needs to change to make it work:

# --------------
# Simulation of a request initiated by not-yet-available 1.11.2 and 1.12.0 releases
# --------------
$ curl 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' --compressed -H 'x-terraform-version: 1.11.2' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version')
HTTP/2 200
content-type: application/json
content-encoding: gzip
last-modified: Mon, 10 Mar 2025 18:34:53 GMT
server: terraform-registry/7c1548ef76fd957bbf48f8cfe4dc019400789afc
strict-transport-security: max-age=31536000; includeSubDomains; preload
date: Wed, 12 Mar 2025 10:33:59 GMT
cache-control: max-age=30, stale-while-revalidate=1800, stale-if-error=31536000, public
vary: Accept-Encoding
x-cache: Hit from cloudfront
via: 1.1 87cff53a3b3c669d865b820d148e2d62.cloudfront.net (CloudFront)
x-amz-cf-pop: HIO50-C2
x-amz-cf-id: 6fEvg6E0d6k2pBlA1Z_QjpI5UkIsGwC4aKKyIyrtIjG6T9BxxeQ_Nw==
age: 220

"0.97.0"

$ curl 'https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions' --compressed -H 'x-terraform-version: 1.12.0' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version')
HTTP/2 200
content-type: application/json
date: Wed, 12 Mar 2025 10:39:47 GMT
cache-control: public, max-age=30, stale-while-revalidate=1800, stale-if-error=31536000
content-encoding: gzip
last-modified: Mon, 10 Mar 2025 18:34:53 GMT
server: terraform-registry/7c1548ef76fd957bbf48f8cfe4dc019400789afc
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 44e82525c613ef192153f7e6e465977a.cloudfront.net (CloudFront)
x-amz-cf-pop: SEA900-P2
x-amz-cf-id: YV5yDghiWkEsnr8WY98Yo0gbvS1MRNhPPqh2Og_N4NZrzulTuGg9lQ==

"0.97.0"

# --------------
# 1.11.1 w/ dummy query param
# --------------
$ curl "https://registry.terraform.io/v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions?test=$(date +%s)" --compressed -H 'x-terraform-version: 1.11.1' -SsD - -o >(jq '.modules[0].versions[] | select(.version == "0.97.0").version')
HTTP/2 200
content-type: application/json
date: Wed, 12 Mar 2025 10:41:07 GMT
cache-control: max-age=30, stale-while-revalidate=1800, stale-if-error=31536000, public
content-encoding: gzip
last-modified: Mon, 10 Mar 2025 18:34:53 GMT
server: terraform-registry/7c1548ef76fd957bbf48f8cfe4dc019400789afc
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Accept-Encoding
x-cache: Miss from cloudfront
via: 1.1 a9b50852e198f506f036f962723ada50.cloudfront.net (CloudFront)
x-amz-cf-pop: SEA900-P2
x-amz-cf-id: ay--oy6MODmfl4CU0D4QrcE4n22x8lAk4HxCu9adwVCbJHiCNxjRYQ==

"0.97.0"

Of course, it may happen that more objects got cached this way, but would it be possible to invalidate /v1/modules/cloudposse/cloudfront-s3-cdn/aws/versions as an interim solution, please?

[jbardin]

Thanks for taking the time to debug that @jwadolowski! That makes total sense; in my quick triage I used v0.11.1 to test the code, but I didn't bother to set the release version for the local build, so the x-terraform-version didn't match!

We'll check with the registry team and see if they can sort out the caching discrepancy.

Thanks!

[justinretzolk]

Similar to an issue with the most recent AWS Provider release: hashicorp/terraform-provider-aws#41763

[jwadolowski]

@jbardin any news from the registry team?

I upgraded to 1.11.2 as soon as it was released and, as anticipated, the issue automatically resolved itself.

Nevertheless, 1.11.1 is still affected and as per previous comment there's more endpoints that got indefinitely cached and serve stale data.

[BannedByHashiCorp]

By the way, a week later, this is still happening for all modules and providers.

[crw]

The Registry team is aware of this issue. This repository is not used or seen by the Registry team, if you wish to file a ticket directly to that team please use their support channel: terraform-registry@hashicorp.com.

I will leave this issue open until resolution just in case there is something that the Terraform core project team needs to do refresh these caches. Thanks.

[crw]

Per the Registry team, this issue should be resolved by EOD tomorrow, Mar 21 (caches should be completely flushed and regenerated by that time). I will close this issue as it has been declared resolved by the relevant team. Thanks!