Regression: `atmos list stacks` became very slow since v1.214.0

[mss]

Describe the Bug

We currently use v1.214.0. I tried to verify that everything still works in v1.228.0 and a simple atmos list stacks became very slow:

v1.214.0

# time atmos list stacks | wc -l
25

real    0m0,434s
user    0m1,053s
sys     0m0,109s

v1.228.0

# time atmos list stacks | wc -l
25

real    0m35,679s
user    0m30,667s
sys     0m2,094s

I think something has changed with deferred template parsing because I had to comment out some code calling !terraform.output which otherwise caused an error

   Error: failed to execute describe stacks: failed to resolve deferred YAML functions in "vars": failed to process YAML function at subnets: failed to render template with datasources: failed to render deferred YAML function value:    
   template: deferred-yaml-function-value:1:89: executing "deferred-yaml-function-value" at <.settings.account.environment>: map has no entry for key "account"                                                                             ```

Expected Behavior

Listing a few stacks shouldn't take half a minute.

Steps to Reproduce

No response

Screenshots

No response

Environment

• Version: v1.228.0

Additional Context

I should be able to do some bisecting to find the version where this was introduced.

This might be the same issue as #2344 even though we don't use locals.

Out atmos.yaml:

base_path: !repo-root '.'

components:
  terraform:
    base_path: 'components/terraform'
    deploy_run_init: true
    init_run_reconfigure: true
    auto_generate_backend_file: true
    workspaces_enabled: false

stacks:
  base_path: 'stacks'
  included_paths:
    - 'site/**/*.yaml'
  excluded_paths:
    - '**/_*.yaml'
  name_template: '{{ .settings.context.namespace }}-{{ .settings.context.tenant }}-{{ .settings.context.environment }}-{{ .settings.context.stage }}'

vendor:
  base_path: 'vendor'

templates:
  settings:
    enabled: true
    evaluations: 3
    sprig:
      enabled: true
    gomplate:
      enabled: true

version:
  check:
    enabled: false
settings:
  telemetry:
    enabled: false

commands: !include 'scripts/commands.yaml'

[84dnnvbdvp-debug]

I think there’s a useful bisect/diagnostic boundary here, though I wouldn’t call it the root cause yet.

Current list stacks routes through ExecuteDescribeStacksScoped, and --process-functions defaults to true. Commit 0a787e6536ebc2e7461d933e321c9684ed6e6d7e (#2892, Aug 13) explicitly wired Stage-3 resolveDeferredYamlFunctions into the standalone describe_stacks processor because that path previously never ran Stage 3. The release boundary is unusually clean: v1.225.0 was published Aug 5 (before that commit); v1.226.0 was published Aug 19 and contains it.

Two tests would narrow this a lot:

1. Compare atmos list stacks on v1.225.0 vs v1.226.0 with the same config.
2. On v1.228.0, compare the normal command with atmos list stacks --process-functions=false.

If the slowdown first appears in v1.226.0 and disabling function processing restores the old timing (and avoids the new !terraform.output resolution error), that would strongly localize the regression to deferred-function evaluation being pulled into the list/describe path.

One caveat: the repo’s Aug 13 Stage-3 fix log also documents a temporary double-execution regression, but specifically notes !terraform.output / !terraform.state duplicate calls were memoized, so I don’t think that known doubling by itself explains ~0.4s → ~35.7s. The more interesting question may be whether a name-only inventory path now resolves remote deferred values at all.

[mss]

The issue was introduced in v1.216.0 (via #2363) where the time jumped to 5 minutes and the output made it obvious that it was pulling data from !terraform.output:

version 1.217.0
real    6m25,977s
user    5m27,231s
sys     0m19,904s

version 1.216.0
real    0m0,418s
user    0m1,019s
sys     0m0,100s

The sitation was improved in v1.222.0 where it was reduced to "only" 30 seconds:

version 1.222.0
real    0m34,336s
user    0m29,629s
sys     0m1,834s

version 1.221.0
real    6m35,948s
user    5m24,738s
sys     0m22,481s

[84dnnvbdvp-debug]

Thanks — that timing rules out my v1.226 localization.

I checked #2363, and it looks like the earlier behavior change is the stronger boundary: that PR added --process-functions to list stacks and intentionally defaulted it to true. Its own description says that flag controls YAML functions including !terraform.output, !terraform.state, !store, etc. So a name/inventory command that previously did not evaluate those functions began doing so by default.

Your measurements put the actual timing discontinuity between v1.216.0 (~0.4s) and v1.217.0 (~6m26s), even though the prose says “introduced in v1.216.0”; either way, #2363 is much earlier than #2892 and matches the observed !terraform.output activity directly.

I would now separate two questions:

1. The original regression: should list stacks eagerly resolve remote YAML functions by default at all? feat(list): --process-templates and --process-functions flags; fix list instances --upload auth #2363 appears to be the source of that behavior.
2. The later v1.221 → v1.222 improvement: I don't have enough source evidence yet to attribute that reduction from ~6m36s to ~34s.

If you haven't already tried it, atmos list stacks --process-functions=false on a current version would still be a useful semantic check: if it restores inventory-speed behavior, that would support changing the default/processing policy for list operations rather than chasing the later Stage-3 path as the origin.

[mss]

84dnnvbdvp-debug you are obviously a bot and not helpful can you please stop commenting on this issue?

[osterman]

(don't know where that bot came from)

[osterman]

Malte S. Stretz (@mss) that's annoying and agree in principle that atmos list stacks should be fast. Here's more likely what is going on.

1. A lot of functions in 1.214.0 didn't properly handle auth, so they simply didn't resolve output correctly. This is why it was "fast"
2. We've fixed all call sites to properly handle atmos auth, so functions like atmos.Component and !terraform.output and !terraform.state call them; then it's "slower" because it works
3. Configs that call terraform.output are the slowest possible; they literally call terraform init (which initializes the terraform provider downloading all providers, state etc). If you can, you should move to !terraform.state and it will be 10x faster.
4. Calling !terraform.state is much faster because we retreive state from supported backends directly; it still results in a network call, which depending on the network connection can be slow. E.g. accessing us-east-1 from EU.

Why do we care? Because atmos list stacks is like a view. You can customize the columns and those columns can contain any part of the stack configuration in the atmos config.

You have a few options: (that can be combined)

1. replace !terraform.output with !terraform.state; also if you are not auth'd, atmos gracefully falls back and will just display <computed> for any value it cannot resolve
2. add flags like --process-functions=false, --process-templates=false, --skip terraform.state; I recognize that's not ideal from a DX perspective

I think the ideal solution is if values are lazy loaded only if displayed in the actual output from list stacks.

Let me know if any of this speeds it up, as it will help us understand where the slowdown exists.

[mss]

Erik Osterman (Cloud Posse) (@osterman) I don't think it is auth related since we don't use that feature at all yet (due to #1950 but maybe that was fixed with the auth improvements you mentioned in the meantime). We'd love to use terraform.state instead of terraform.output but are blocked by #1384 (we still have a ticket deep in our own backlog to have a shot at trying to implement that).

I agree that lazy loading the data and not templating the rest of the manifests is probably the best and only solution to get this fixed. Maybe as a hotfix the plain calls to atmos list stacks and atmos list components -s ... could internally set a flag to skip these expensive calls? At least if templating in stack and component names is still not supported.

I tried the various flags and are a bit surprised by the results: I only get back the old speed by specifying --process-templates=false. I'd expect that --process-functions=false or --skip terraform.output would have the same effect but those don't change a thing about the 30 seconds. Is it then really the templating which makes things that slow? We do indeed have the issue in another project which has hundreds of stacks (and is still pinned on an even older version of atmos) by now that atmos describe stacks got extremely slow over time.

[mss]

Erik Osterman (Cloud Posse) (@osterman) I think I found something. We use terraform.output quite a lot. I ran atmos --logs-level debug list stacks (without --process-functions=false) for both v1.221.0 (where it still took 5+ minutes) and v1.228.0. When I grep for terraform.output in both cases I see that the old version called it 385 times whereas the current version calls it only 4 times. Executing v1.221.0 with --process-functions=false actually gets its processing time and calls down to the same as v1.228.0.

Nice. But why is it still calling the function 4 times? Why is it calling it at all?

It looks like it does so only for two of our stacks. Even though we have other stacks which are very similar since they use the same abstract base components. I am trying to find out what the difference is.

[mss]

Erik Osterman (Cloud Posse) (@osterman) I found the place where the function was still called. It isn't !terraform.output but a call to atmos.Component, ie. not a YAML function but a template function:

# ...
components:
  terraform:
    # ...
    argocd-application:
      # ...
      vars:
        # ...
        helm_parameters:
          releaseName: "vault"
          parameters:
            - name: "vault_init.secretName"
              value: "eks/{{ (atmos.Component \"eks/cluster\" \"eks-svc-euc1-1-production\").outputs.cluster_name }}/cluster/vault-root-token"
            - name: "vault_init.kmsKeyId"
              value: !terraform.output vault-infra kms_seal_key_id
            - name: "vault.server.kmsKeyId"
              value: !terraform.output vault-infra kms_seal_key_id
            - name: "vault.server.ingress.hosts[0].host"
              value: '{{ .settings.vault.host }}'

I think I can avoid this call.

Lazy template evaluation is probably the only possible solution here.