feat: provider cache proxy for parallel terraform init (PROVIDER_CACHE) - #6840
Open
sakkiii wants to merge 11 commits into
Open
feat: provider cache proxy for parallel terraform init (PROVIDER_CACHE)#6840sakkiii wants to merge 11 commits into
sakkiii wants to merge 11 commits into
Conversation
Signed-off-by: sakkiii <s@sakkiii.in>
Signed-off-by: sakkiii <s@sakkiii.in>
Add an opt-in local caching proxy for Terraform providers, modeled on Terragrunt's provider cache server. When enabled with --provider-cache, Atlantis starts a small HTTP server on localhost that speaks the Terraform Provider Registry Protocol and points terraform at it via a host block in a generated ~/.terraformrc CLI config file. The many parallel `terraform init` runs Atlantis executes across workspaces and pull requests then fetch providers through the proxy, which downloads each provider archive from the upstream registry exactly once (concurrent requests for the same artifact are de-duplicated with singleflight), caches it on disk, and serves the cached copy to every subsequent request. Archive bytes are served verbatim so terraform's checksum and GPG-signature verification is unaffected. New flags: --provider-cache, --provider-cache-dir, --provider-cache-port, --provider-cache-registry-hosts. Signed-off-by: sakkiii <s@sakkiii.in>
Signed-off-by: sakkiii <s@sakkiii.in>
sakkiii
changed the title
sakkiii
marked this pull request as ready for review
September 3, 2026 15:36
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Restrict provider metadata requests to the configured registry allowlist:
handleVersions/handleDownload now reject any {host} not in --provider-cache-
registry-hosts and use the trusted, configured spelling of the host, so the
proxy cannot be used to reach arbitrary hosts. Path segments are already
url.PathEscape'd against a trusted base, so the destination host can't be
altered.
- Make cached artifact filenames the bare hex SHA-256 of the URL (fixed charset,
no separators/dots), wrapped in filepath.Base, removing any user-derived
component from on-disk paths.
- Add a test asserting unconfigured registry hosts are refused.
The remaining artifact-download request is bound by the per-process HMAC
signature (only proxy-generated URLs are fetched) plus an https/loopback scheme
check; fetching the registry-provided download URL is the endpoint's purpose.
Signed-off-by: sakkiii <s@sakkiii.in>
- Modernize: use strings.SplitSeq, range-over-int, and WaitGroup.Go. - Suppress the three gosec taint findings (G703 path traversal on the hash-named cache file served via http.ServeFile; G704 SSRF on the artifact download) with #nosec directives that document why each is safe: the cache file name is a bare hex SHA-256 that cannot escape the cache dir, and the downloaded URL is HMAC-signed by this process and scheme-restricted before it reaches download(). Signed-off-by: sakkiii <s@sakkiii.in>
… SSRF
Rework the artifact endpoint so it carries no request-supplied URL: Terraform
now receives coordinate-addressed artifact URLs
(/artifact/{host}/{ns}/{type}/{version}/{os}/{arch}/{kind}) and the proxy
re-resolves the real download location from the trusted registry's own
download-metadata response. Combined with a strict per-segment character
allowlist (segmentPattern) and the existing registry-host allowlist, no
attacker-controlled value from the incoming request reaches an outbound request
target or an on-disk path.
Validated with CodeQL (go-security-extended): 0 request-forgery and 0
path-injection alerts in the package (previously 3 critical request-forgery +
path-injection). The HMAC-signed-URL machinery is removed as it is no longer
needed. gosec's coarser taint pass is quieted with justified #nosec directives.
Also fix an MD049 markdownlint error in the provider-cache docs.
Signed-off-by: sakkiii <s@sakkiii.in>
Alpine 3.23's repository replaced curl-8.20.0-r0 with curl-8.22.0-r0 and dropped the old version, so `apk add curl=8.20.0-r0` no longer resolves and the image build fails. Bump the pin to the currently available version. Verified with `apk add --simulate` against alpine:3.23.5 that the full package set resolves; the other pins are unchanged and still current. Signed-off-by: sakkiii <s@sakkiii.in>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
what
--provider-cache.localhostthat speaks the Terraform Provider Registry Protocol, and points Terraform at it via ahostblock in a generated~/.terraformrcCLI config file (merged with the existing TFEcredentialsblock).SHA256SUMS/ signature download URLs back through itself, so it downloads each artifact from the origin once, de-duplicates concurrent requests (singleflight), caches on disk, and serves the cached copy to every subsequent request.--provider-cache,--provider-cache-dir,--provider-cache-port,--provider-cache-registry-hosts.why
terraform initcommands in parallel (across workspaces, projects and PRs), each process independently downloads the same providers from the upstream registry — wasting bandwidth, hitting registry rate limits, and racing on the shared plugin-cache dir (text file busy/ partial-download failures).--use-tf-plugin-cache: the plugin cache lets one process reuse an installed provider; this proxy deduplicates the downloads across many parallel processes.hostservice-discovery override (plain HTTP to loopback) rather thannetwork_mirror(which requires HTTPS/certs) — the same approach Terragrunt uses.tests
go build,go vet,gofmt, and the affected package tests pass on Go 1.26.references
parallel_plan/parallel_apply— concurrent downloads collapse to one, removing thetext file busy/ partial-cache race)usePluginCacheconfiguration option to the configuration files. #3547, refs TF_PLUGIN_CACHE_DIR is not present in custom steps #5583 (alternative caching mechanism)architecture decision record
ADR: