Atmos scaffold dynamic file generation via for_each

[Jorrit Elfferich (jorrite)]

Hey folks 👋

atmos scaffold's spec.files[].when: can only gate a fixed, enumerable set of files declared up front, it can skip a file, but it can't multiply one. That leaves a gap: there's no way to generate one file per item in an unbounded or structured answer (e.g. "one stacks/<env>.yaml per environment the user typed," or "one file per region × environment in a nested answer").

This gap is already defined in the current PRD for scaffold 👉 see here.

I'd like to take a stab at this, so I put together a proposal for a spec.files[].for_each: construct that closes this gap, staying entirely inside the existing spec.files[] interface. What it would look like:

spec:
  files:
    - path: environment.yaml
      target: "stacks/{{ .env.value }}.yaml"
      for_each:
        - as: env
          in: answers.environments
          split: ","

for_each iterates a list, a map, or (only when needed) a delimited free-text string — and it chains: a later step can iterate something nested inside an earlier step's bound item, so one entry can express a cross-product like "one file per environment within each region."

Didn't exactly knew where to put it, so I forked off and made some adjustments to the scaffold PRD to serve as a proposal:

📄 docs/prd/atmos-scaffold.md — see "Dynamic File Generation (for_each) — Proposal"
🔀 Diff against cloudposse/atmos:main

FWIW, for truly complex shapes where you'd have a matrix of regions and environments (potentially not even a full matrix if you only want multi-region on some environments but not all), we are already piping through those shapes by just baking a .atmos/scaffold.yaml prior to kicking off the scaffold command. Not ideal, but works flawlessly.

I have a working version of the for_each implementation already to just verify if it was even at all feasible, but I would love thoughts before I sink more time into a final implementation, especially on:

• whether spec.files[] is the right home for this vs. a separate top-level construct
• the in: convention (answers.<path> for the first step, unprefixed for a chained step referencing an earlier binding)
• appetite for this at all vs. other priorities.

If the direction looks good, happy to work on it!

[Erik Osterman (Cloud Posse) (osterman)]

Proposal: use matrix, not for_each

We prefer the matrix step over the for_each design.

Why

Atmos already has a matrix step for workflows and custom commands:

steps:
  - name: test-matrix
    type: matrix
    matrix:
      os: [linux, darwin]
      go: ['1.22', '1.23']
    steps:
      - name: test
        type: shell
        command: make test OS={{ .matrix.os }} GO_VERSION={{ .matrix.go }}

matrix: maps an axis name to a list of values. Atmos iterates over every combination of the axes. Each combination binds to .matrix.<axis>. Users already know this pattern from workflows.

for_each adds new terms: as:, in:, split:, and chaining across steps. This looks like a loop. A loop invites more loop features: nested loops, conditions inside the loop, computed sources. We want to avoid this.

matrix: has a fixed shape. It iterates over a product of known axes. It has no named variable to chain. It has no place for an if. It cannot grow into a general loop.

Changes to the shipped matrix step

Axis values must come from user answers, not only fixed lists. We resolve this in the template string, with Sprig's splitList function. We do not add a split: key.

spec:
  files:
    - path: environment.yaml
      target: 'stacks/{{ .matrix.env }}.yaml'
      matrix:
        env: '{{ answers.environments | splitList "," }}'

Multiple axes work the same way as in workflows:

    - path: workflow.yaml
      target: 'workflows/{{ .matrix.env }}-{{ .matrix.action }}.yaml'
      matrix:
        env: '{{ answers.environments | splitList "," }}'
        action: [plan, apply]

Filtering one axis with when:

when: and matrix: sit side by side on a file entry. No new field is needed for the file itself:

    - path: environment.yaml
      target: 'stacks/{{ .matrix.os }}.yaml'
      matrix:
        os: [linux, darwin]
      when: 'matrix.os == "linux"'

This generates stacks/linux.yaml only. It skips the darwin combination.

Dependent values: use when:, not chaining

Some cases need dependent values. Example: each region has a different list of valid environments.

Atmos does not build the full matrix as one list first. Atmos iterates over each combination one at a time. For each combination, when: runs in the context of that combination — the current region and env are already bound. If when: returns false, Atmos skips that file and moves to the next combination.

    - path: templates/deploy.yaml
      target: 'deploy/{{ .matrix.region }}/{{ .matrix.env }}.yaml'
      matrix:
        region: '{{ keys answers.regions }}'
        env: [dev, staging, prod]
      when: 'matrix.env in answers.regions[matrix.region].environments'

This needs one change to pkg/condition: add a Matrix field to condition.Context. Add a matrix variable to the CEL environment. Set this field fresh for each combination, not once for the whole run.

when: uses CEL, and we already use CEL

when: is not custom syntax. It is CEL (Common Expression Language). Atmos already uses CEL for hook events, custom command steps, and scaffold fields and files. Adding matrix.* to when: does not add a new language. It adds one variable to a system we already use.

CEL is a standard choice outside Atmos too. Kubernetes uses it for ValidatingAdmissionPolicy. Envoy uses it for RBAC and routing rules. Firebase uses it for security rules. CEL has no loops. It has no unbounded recursion. It has no custom functions. Every CEL expression ends.

We want the same limits for matrix:. A bounded expression language for conditions, plus a bounded iteration operation, keeps the config declarative. for_each breaks that pattern.

Implementation note

This design needs one change to pkg/condition: add a Matrix field to condition.Context, add a matrix variable to the CEL environment, and set it fresh per combination. Whoever picks up this work should make that change as part of it. celMentionsIdentifier already checks for identifier use in CEL expressions and can extend to require a matrix: block whenever when: uses matrix.*.

[Jorrit Elfferich (jorrite)]

Appreciate the feedback! I didn't consider matrix and was maybe a bit too hung up on what was described in the PRD with "for each"-like behavior so went in that direction. I think matrix: + when: is indeed the better home for this. Reusing the interface that people already know from workflows, and keeping CEL as the only place any logic lives, is sound.

Two things I'd like to nail down, though:

1. Content, not just target: and when:. Every example in your write-up shows .matrix.<axis> inside target: and/or when:. Is it also meant to be available inside the template file's own body when it renders? I would assume it is, so that everything receives the same data map, and we could use those values inside the file itself if needed. This behavior isn't called out anywhere and I'd rather have it confirmed than assumed. The shipped workflow matrix step isn't a precedent here as it only threads .matrix.<axis> into a handful of the step's own fields (command, stack, timeout, workdir, env), it never renders a separate file per combination like scaffold would, so there's no existing behavior to lean on for this specifically.

2. Computed axes, concretely. Take three environments, only production being multi-region:

environments:
  dev:
    regions:
      us-east-1: {...}
  staging:
    regions:
      us-east-1: {...}
  production:
    regions:
      us-east-1: {...}
      us-west-2: {...}

With matrix:, you'd fill out the whole matrix — every environment × every region that appears anywhere — and let when: skip the combinations that don't apply (so for the example, these would be skipped: dev+us-west-2, staging+us-west-2), but how would you then get the full flattened region list? Looking at what sprig and Gomplate offer, we could probably use the jq function from Gomplate, and it would look like:

matrix:
  environments: '{{ answers.environments | jq "keys | sort" }}'
  regions: '{{ answers.environments | jq "[.[].regions | keys[]] | unique | sort" }}'
when: 'matrix.regions in answers.environments[matrix.environments].regions'

That would work well. Thoughts on that?

[Erik Osterman (Cloud Posse) (osterman)]

Content, not just target: and when:. Every example in your write-up shows .matrix. inside target: and/or when:. Is it also meant to be available inside the template file's own body when it renders?

Yes, let's ensure it's also available to the template.

but how would you then get the full flattened region list?

Yes, what you propose could work. Ideally we could expose some happy-path, simple method as well, to avoid complex expressions.

matrix:
  environments: '{{ keys answers.environments }}'
  regions: '{{ keys answers.environments "regions" }}'
when: 'matrix.regions in answers.environments[matrix.environments].regions'

Here, keys answers.environments "regions" means: collect the keys under every environment’s regions, flatten them, and remove duplicates. Sorting could be deterministic by default or added with | sort.

This looks pretty clean to me.