You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,6 +15,8 @@ and then passes status updates of those Runs back to the Merge/Pull Request in t
15
15
16
16
For MRs with multiple workspaces, TFBuddy tracks each workspace independently and can automatically clean up old plan/apply comments, keeping only the most recent one per workspace and action type. Set `TFBUDDY_DELETE_OLD_COMMENTS` to enable this.
17
17
18
+
Each touched workspace is dispatched onto a dedicated NATS JetStream queue and processed independently, so a single MR with many workspaces can no longer hold the webhook subscriber past `AckWait` (which previously caused JetStream redeliveries and duplicate TFC runs). The fan-out is gated by `TFBUDDY_WORKSPACE_FANOUT_ENABLED` (default `true`) so it can be turned off at runtime if needed. The TFC API client is rate-limited (`TFBUDDY_TFC_RATE_LIMIT_RPS` / `_BURST`) so concurrent workers stay under TFC's documented per-token limit.
Copy file name to clipboardExpand all lines: docs/architecture.md
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,6 +27,24 @@ Example of how an error is reported
27
27
A more detailed error message is provided if the error occurred within TF Buddy. If the error occurred during terraform plan or terraform apply it will not contain a detailed message. Instead the run link will take you to the Terraform workspace where you can debug the error.
28
28
29
29
30
+
### Workspace Fan-Out
31
+
32
+
When an MR/PR touches several Terraform workspaces, TFBuddy splits the work onto a dedicated NATS JetStream queue and processes each workspace independently. The MR/PR webhook subscriber clones the repo once, evaluates which workspaces are blocked by changes on the target branch, and publishes one fan-out message per remaining workspace. A separate workspace worker drains the queue, with each delivery getting its own `AckWait` window — a slow workspace can no longer hold the parent message past `AckWait` and trigger a redelivery (which previously caused duplicate runs).
33
+
34
+
Per-workspace state (discussion ID, root note ID) is local to the worker invocation, so concurrent deliveries cannot leak IDs across workspaces.
35
+
36
+
The fan-out is gated by `TFBUDDY_WORKSPACE_FANOUT_ENABLED` (default `true`). Setting it to `false` falls back to the legacy inline per-MR loop, useful if you need to roll back at runtime without redeploying.
37
+
38
+
Each fan-out message carries the upstream webhook delivery ID (`X-GitHub-Delivery` for GitHub, `X-Gitlab-Event-UUID`/`Idempotency-Key` for GitLab). That ID combined with the workspace name is used as the JetStream dedup key, so a single webhook fanning out to N workspaces produces N distinct keys, while a legitimate retrigger (a new push or comment within the dedup window) carries a fresh delivery ID and is *not* swallowed by the dedup window.
39
+
40
+
Target-branch evaluation runs once in the publisher, so the MR-level "could not evaluate target branch" warning is posted at most once per delivery. Each worker then re-clones the repo to upload the workspace directory to TFC, so an MR touching N workspaces still does N+1 clones in fan-out mode. On large monorepos this measurably increases git egress and per-message latency; mitigate via `TFBUDDY_GITHUB_CLONE_DEPTH` / `TFBUDDY_GITLAB_CLONE_DEPTH`, or disable fan-out (`TFBUDDY_WORKSPACE_FANOUT_ENABLED=false`) and accept the AckWait risk.
41
+
42
+
The workspace stream exposes the same `consumers >= 1` liveness check as the other streams, wired into `/live` so a stuck or unsubscribed worker is caught by Kubernetes probes rather than silently piling up unprocessed messages.
43
+
44
+
### TFC API Rate Limiter
45
+
46
+
The TFC API client wraps its HTTP transport with a token-bucket rate limiter (`TFBUDDY_TFC_RATE_LIMIT_RPS`, default `30`; burst `TFBUDDY_TFC_RATE_LIMIT_BURST`, default `30`) so concurrent workspace workers cooperatively stay under TFC's documented per-token limit. The client deliberately has no top-level `Timeout`: `ConfigurationVersions.Upload` streams the cloned repo and a fixed cap would truncate slow uploads on large repos. Per-call deadlines flow through `context.Context`.
47
+
30
48
### Webhook Management
31
49
32
50
At Zapier we automate the creation of all relevant webhooks by leveraging Terraform to create them. The example below is a resource we use to hook up a Gitlab project and a Terraform Cloud workspace to TF Buddy.
Copy file name to clipboardExpand all lines: docs/contributing.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,6 +108,12 @@ If you're using minikube with Tilt we recommend following this [guide](https://g
108
108
109
109
We use Earthly to simplify our CI/CD process with TF Buddy. This also simplifies testing changes locally before pushing them up to ensure your PR will pass all required checks. The best command to run is `earthly +unit-test` this will pull all the required dependencies (including any new ones that you have added). It will then run [go vet](https://pkg.go.dev/cmd/vet), and if those pass it will run `go test` with race detection enabled. You can also always run these commands directly `go test -race ./...` will run all tests in the repo with race detection enabled. Please ensure that `earthly +unit-test` is passing before opening a PR.
110
110
111
+
Always run with `-race` when changing the trigger or TFC API code — those paths are concurrent (workspace fan-out, shared rate limiter):
112
+
113
+
```console
114
+
go test -race ./pkg/tfc_trigger/... ./pkg/tfc_api/...
115
+
```
116
+
111
117
#### Unit Tests
112
118
113
119
We use [gomock](https://github.com/golang/mock) to simplify down some of our unit tests specifically in `tfc_trigger_tests.go` where a lot of our functionality needs to be tested. To reduce the burden of testing a new feature you can leverage the `TestSuite` in `pkg/mocks/helpers.go`. When you call `CreateTestSuite` it will return a set of stubs that should let you test most operations. To override stubs that are created within the TestSuite define them after calling `CreateTestSuite` but before calling `InitTestSuite`. This works because gomock will respect the order that stubs/mocks are defined which lets you override an EXPECT defined in TestSuite. This can be seen in this example:
Copy file name to clipboardExpand all lines: docs/usage.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -68,6 +68,10 @@ The full list of supported environment variables and flags is described below:
68
68
|`TFBUDDY_GITHUB_REPO_ALLOW_LIST`|`--github-repo-allow-list`|Comma-separated GitHub repository allow list prefixes.||
69
69
|`TFBUDDY_GITHUB_CLONE_DEPTH`|`--github-clone-depth`|Git clone depth to use for GitHub merge request checkouts. Zero means full history.|`0`|
70
70
|`TFBUDDY_GITLAB_CLONE_DEPTH`|`--gitlab-clone-depth`|Git clone depth to use for GitLab merge request checkouts. Zero means full history.|`0`|
71
+
|`TFBUDDY_WORKSPACE_FANOUT_ENABLED`|`--workspace-fanout-enabled`|Enable per-workspace JetStream fan-out (one NATS message per workspace) to keep AckWait windows scoped per workspace. When disabled, TFBuddy falls back to the inline per-MR loop.|`true`|
72
+
|`TFBUDDY_WORKSPACE_JETSTREAM_REPLICAS`|`--workspace-jetstream-replicas`|JetStream replica count for the workspace-trigger stream. Use 1 for single-node NATS or local dev; set to your NATS cluster size (often 3) in production for durability.|`1`|
73
+
|`TFBUDDY_TFC_RATE_LIMIT_RPS`|`--tfc-rate-limit-rps`|Client-side rate limit (requests per second) for the Terraform Cloud API. Tuned to match TFC's documented per-token limit and prevent 429s when many workspaces are triggered concurrently.|`30`|
74
+
|`TFBUDDY_TFC_RATE_LIMIT_BURST`|`--tfc-rate-limit-burst`|Burst capacity for the TFC API token-bucket rate limiter.|`30`|
71
75
<!-- END GENERATED CONFIGURATION -->
72
76
73
77
For sensitive environment variables use `secrets.envs` which can contain a list of key/value pairs
{key: KeyGithubCloneDepth, defaultValue: 0, description: "Git clone depth to use for GitHub merge request checkouts. Zero means full history."},
85
93
{key: KeyGitlabCloneDepth, defaultValue: 0, description: "Git clone depth to use for GitLab merge request checkouts. Zero means full history."},
94
+
{key: KeyWorkspaceFanoutEnabled, defaultValue: true, description: "Enable per-workspace JetStream fan-out (one NATS message per workspace) to keep AckWait windows scoped per workspace. When disabled, TFBuddy falls back to the inline per-MR loop."},
95
+
{key: KeyWorkspaceJetStreamReplicas, defaultValue: 1, description: "JetStream replica count for the workspace-trigger stream. Use 1 for single-node NATS or local dev; set to your NATS cluster size (often 3) in production for durability."},
96
+
{key: KeyTFCRateLimitRPS, defaultValue: 30, description: "Client-side rate limit (requests per second) for the Terraform Cloud API. Tuned to match TFC's documented per-token limit and prevent 429s when many workspaces are triggered concurrently."},
97
+
{key: KeyTFCRateLimitBurst, defaultValue: 30, description: "Burst capacity for the TFC API token-bucket rate limiter."},
0 commit comments