Unclear lag between providers releasing a new version and it appearing in the registry

[joe-a-dbt]

Thank you for reporting an issue.

I am trying to figure out why https://github.com/fivetran/terraform-provider-fivetran/releases/tag/v1.9.31 (which was released over an hour ago at the time I'm writing this) did not yet appear in the OpenTofu registry, which is still showing v1.9.30 as the latest. https://github.com/fivetran/terraform-provider-fivetran/releases/tag/v1.9.31 was released at 10:47AM CDT but 5404864 did not run until 12:03PM CDT (76 minutes later). Additionally, https://search.opentofu.org/provider/fivetran/fivetran/latest is still showing 1.9.30 as the latest as of 12:27PM CDT (100 minutes later). This appears to be a significant variance from https://github.com/opentofu/registry/blob/main/PROCEDURES.md#user-complaint-provider-version-is-not-available for maintainer troubleshooting, which sets a 30 min expectation.

Additionally, beyond this immediate instance I'm flagging, I also want to flag the need for improving the documentation in the README. It would be great if the README or a more use-facing document than the PROCEDURES set a clear expectation for how long it should take between a provider releasing a new version and the OpenTofu registry being updated.

OpenTofu Version

N/A

OpenTofu Configuration Files

N/A

[joe-a-dbt]

Looks like 1.9.31 showed up in the last few minutes. But I still think some investigation is warranted about why it took so long and clarifying the documentation.

[Yantrio]

I understand the frustration here of the delay, it's something that we're discussing internally. I'd love to explain a little bit more about what we're seeing here though.

Intended flow

The registry works on a three-tiered system of GitHub Actions. These are in charge of three data-sets.

• Registry metadata in this repository
• API Data in cloudflare r2
• search.opentofu.org documentation scraping

All 3 of these tiers act independantly through GitHub Actions across both github.com/opentofu/registry and github.com/opentofu/registry-ui

The first tier aims to run every 15 minutes , and will check github repositories for new releases and commit that to this repository. This is what you are seeing generating this commit.
Action: https://github.com/opentofu/registry/blob/main/.github/workflows/bump-versions.yml#L11
Cron schedule: 0,15,30,45 * * * *

The second tier aims to also run every 15 minutes, but offset by 5 minutes. This will take the registry metadata, generate all the possible responses for the registry API (used by opentofu) and uploads that data to cloudflare R2.
Action: https://github.com/opentofu/registry/blob/main/.github/workflows/generate-and-deploy-recent.yml#L6
Cron schedule: 5,20,35,50 * * * *

The third tier we deem to be less critical and is to scrape the documentation for new versions that we have detected. This aims to run every 15 minutes like the others. In this, we will git clone each repository that has had a new release, and scrape the documentation files from the repository to upload to our cloudflare R2. This is what drives search.opentofu.org.
Action: https://github.com/opentofu/registry-ui/blob/f735bd9b6e0a50d03aa5c45e672e3a9d317cc349/.github/workflows/index.yml#L6
Cron schedule: "*/15 * * * *

The three tiers together do induce a slight delay, but allow us to define 3 levels of criticality and stability.

We also designed this to be run in GitHub actions because this gives end-users full visibility into what's happening.

What is happening lately

I'm not here to make excuses for other companies, but we've noticed a large degredation in how often our GitHub actions are being run. Since the inception of the registry until quite recently, the intended timings were working extremly well. However due to GitHub and it's stability, we're seeing that actions are being far less frequently.

For example, the most recent runs of the first tier which you can find here: https://github.com/opentofu/registry/actions/workflows/bump-versions.yml has some gaps as large as 92minutes, and as short as 24 minutes. Out of the intended ~70 runs we expected to do today for this tier alone, GitHub has only executed 16.

Because this is happening in 3 tiers, we're seeing the effects of stacking tolerances and this is causing a delay in our ingestion and the time to which a provider or module version is available.

I hope you understand our issues and that we are discussing the best path forward to improving this.

[joe-a-dbt]

Makes sense. And honestly if you all want to close this after documenting in the README that it can take up to 2-3 hours for newly released providers to show up, that's an acceptable response to close this issue as far as I'm concerned. Obviously I'd love if it could be faster, but my primary concern is definitely the expectation mismatch and lack of documentation.