fix: YAML functions fail with templated args when custom delimiters include quotes

docs/fixes/2026-02-20-terraform-state-custom-delimiters.md

@@ -0,0 +1,214 @@
+# Fix YAML Functions Fail with Templated Arguments When Custom Delimiters Include Quotes
+
+**Date:** 2026-02-20
+
+**Related Issue:** [GitHub Issue #2052](https://github.com/cloudposse/atmos/issues/2052) — `!terraform.state` fails with
+`yaml: line NNN: did not find expected key` when custom delimiters include quotes (e.g., `["'{{", "}}'"]`).
+
+**Affected Atmos Version:** v1.205.0+
+
+**Severity:** Medium — users with custom delimiters containing single-quote characters cannot use templated arguments
+in any YAML function, forcing them to use only static arguments.
+
+**Affected YAML Functions:** All Atmos YAML functions that accept template arguments are affected:
+- `!terraform.state` — Terraform state lookups
+- `!terraform.output` — Terraform output lookups
+- `!store` / `!store.get` — Store value lookups
+- `!env` — Environment variable lookups
+- `!exec` — Shell command execution
+- `!include` / `!include.raw` — File inclusion
+- `!random` — Random value generation
+- `!template` — Template evaluation (when containing single quotes)
+
+Functions without arguments (`!cwd`, `!repo-root`, `!aws.*`) are not directly affected since they
+produce no YAML quoting conflicts, but the fix handles them generically if they were to contain
+single-quote characters in the future.
+
+## Background
+
+Atmos supports custom template delimiters via `templates.settings.delimiters` in `atmos.yaml`. Some users configure
+delimiters that include single-quote characters to avoid conflicts with YAML syntax:
+
+```yaml
+templates:
+  settings:
+    delimiters:
+      - "'{{"
+      - "}}'"
+```
+
+With these delimiters, template expressions look like `'{{ .stack }}'` instead of `{{ .stack }}`.
+
+## Symptoms
+
+```yaml
+# Any YAML function with a templated argument fails:
+vars:
+  test: !terraform.state vpc '{{ .stack }}' vpc_id
+  out: !terraform.output vpc '{{ .stack }}' vpc_id
+  data: !store my-store '{{ .stack }}' key
+  cmd: !exec echo '{{ .stack }}'
+```
+
+```text
+$ atmos describe component <component> -s <stack>
+yaml: line NNN: did not find expected key
+```
+
+Static arguments work fine:
+```yaml
+# This works:
+vars:
+  test: !terraform.state vpc foo
+```
+
+## Root Cause
+
+The bug is in the template processing pipeline, specifically in how YAML serialization interacts with custom
+delimiter characters. The root cause is **not specific to `!terraform.state`** — it affects ALL YAML functions
+because the issue is in the YAML encoding layer, not the function execution layer.
+
+### Processing Pipeline
+
+1. **YAML loading**: Custom tags like `!terraform.state` are converted to plain string values
+   (e.g., `"!terraform.state vpc '{{ .stack }}' vpc_id"`).
+2. **YAML serialization**: The component section map is serialized to a YAML string via `ConvertToYAML`.
+3. **Template processing**: The Go template engine processes the YAML string with custom delimiters.
+4. **YAML parsing**: The result is parsed back to a map.
+
+### The Conflict
+
+At step 2, the yaml.v3 encoder must quote strings that start with `!` (YAML's tag indicator). It chooses
+**single-quoted style**, which escapes internal single quotes by doubling them (`''`):
+
+```yaml
+# Input string: !terraform.state vpc '{{ .stack }}' vpc_id
+# YAML-encoded (single-quoted):
+test: '!terraform.state vpc ''{{ .stack }}'' vpc_id'
+```
+
+This same escaping happens for ALL YAML functions since they all start with `!`:
+
+```yaml
+# All of these get single-quoted with '' escaping:
+test: '!terraform.output vpc ''{{ .stack }}'' vpc_id'
+data: '!store my-store ''{{ .stack }}' key''
+cmd: '!exec echo ''{{ .stack }}'''
+```
+
+At step 3, the Go template engine with custom delimiters `'{{` and `}}'` scans the raw YAML text
+looking for the delimiter patterns. It finds `'{{` within the `''{{` sequence (where the first `'` is
+YAML's escape character, and the second `'` is the start of the delimiter).
+
+After template replacement (e.g., `'{{ .stack }}'` → `nonprod`), the YAML string becomes:
+
+```yaml
+test: '!terraform.state vpc 'nonprod' vpc_id'
+```
+
+This is **invalid YAML** — the unescaped single quotes inside the single-quoted string break the parser,
+producing the `did not find expected key` error.
+
+### Why Default Delimiters Work
+
+With default delimiters `{{` and `}}`, the YAML-escaped string is:
+
+```yaml
+test: '!terraform.state vpc ''{{ .stack }}'' vpc_id'
+```
+
+The template engine finds `{{ .stack }}` (no quotes in the delimiter pattern), and after replacement:
+
+```yaml
+test: '!terraform.state vpc ''nonprod'' vpc_id'
+```
+
+This is **valid YAML** — `''` is the proper escape for a single quote in a single-quoted string.
+
+## Fix
+
+### Approach
+
+When custom delimiters contain single-quote characters, use **double-quoted YAML style** for string values
+that contain single quotes. Double-quoted YAML strings don't escape single quotes, preserving the delimiter
+pattern literally:
+
+```yaml
+# Double-quoted (no single-quote escaping):
+test: "!terraform.state vpc '{{ .stack }}' vpc_id"
+out: "!terraform.output vpc '{{ .stack }}' vpc_id"
+data: "!store my-store '{{ .stack }}' key"
+cmd: "!exec echo '{{ .stack }}'"
+```
+
+After template replacement:
+
+```yaml
+test: "!terraform.state vpc nonprod vpc_id"
+out: "!terraform.output vpc nonprod vpc_id"
+data: "!store my-store nonprod key"
+cmd: "!exec echo nonprod"
+```
+
+This is **valid YAML** — the double quotes still surround the entire string.
+
+### Implementation
+
+Added `ConvertToYAMLPreservingDelimiters` function that:
+
+1. Checks if custom delimiters conflict with YAML single-quote escaping.
+2. If so, marshals to a `yaml.Node` tree (instead of using the default encoder).
+3. Walks the node tree and forces `yaml.DoubleQuotedStyle` for scalar nodes containing single quotes.
+4. Encodes the modified node tree to YAML string.
+
+The fix is **generic** — it operates at the YAML serialization level and handles ALL scalar values
+containing single quotes, regardless of which YAML function prefix they use. This means any future
+YAML functions will also be automatically protected.
+
+### Files Changed
+
+| File | Change |
+|------|--------|
+| `pkg/utils/yaml_utils.go` | Add `ConvertToYAMLPreservingDelimiters`, `delimiterConflictsWithYAMLQuoting`, `ensureDoubleQuotedForDelimiterSafety` |
+| `internal/exec/utils.go` | Use `ConvertToYAMLPreservingDelimiters` in template processing pipeline |
+| `internal/exec/describe_stacks.go` | Use `ConvertToYAMLPreservingDelimiters` in all 3 template processing sections |
+| `internal/exec/terraform_generate_varfiles.go` | Use `ConvertToYAMLPreservingDelimiters` in template processing |
+| `internal/exec/terraform_generate_backends.go` | Use `ConvertToYAMLPreservingDelimiters` in template processing |
+
+### Tests
+
+**Unit tests** (`pkg/utils/yaml_utils_delimiter_test.go`):
+- `TestDelimiterConflictsWithYAMLQuoting` — 8 subtests for delimiter conflict detection
+- `TestEnsureDoubleQuotedForDelimiterSafety` — 6 subtests for node style modification
+- `TestConvertToYAMLPreservingDelimiters` — 10 subtests including:
+  - Preserves single-quote delimiters in YAML function values
+  - Falls back to standard encoding for default delimiters
+  - Preserves all values correctly after double-quoting
+  - Template replacement produces valid YAML with custom delimiters
+  - Demonstrates that standard encoding breaks with custom delimiters
+  - Handles nested maps and lists with YAML function values
+- `TestAllYAMLFunctionsPreservedWithCustomDelimiters` — 12 subtests verifying every YAML function prefix:
+  - `!terraform.state`, `!terraform.output`, `!store`, `!store.get`, `!env`, `!exec`,
+    `!template`, `!include`, `!include.raw`, `!repo-root`, `!cwd`, `!random`
+- `TestAllYAMLFunctionsTemplateReplacementWithCustomDelimiters` — 9 subtests simulating full
+  template processing pipeline (serialize → replace → parse) for each YAML function
+- `TestStandardEncodingBreaksAllYAMLFunctionsWithCustomDelimiters` — 18 subtests (9 functions × 2)
+  proving both that standard encoding breaks AND delimiter-safe encoding works for each function
+
+**Integration tests** (`tests/yaml_functions_custom_delimiters_test.go`):
+- `TestTerraformStateWithCustomDelimiters` — 3 subtests:
+  - Regular templates with custom delimiters
+  - `!terraform.state` with static arguments
+  - `!terraform.state` with templated stack argument (core issue #2052)
+- `TestCustomDelimitersTemplateProcessing` — 1 subtest for settings template resolution
+
+Run with:
+```bash
+go test ./pkg/utils/ -run TestConvertToYAMLPreserving -v
+go test ./pkg/utils/ -run TestDelimiterConflicts -v
+go test ./pkg/utils/ -run TestEnsureDoubleQuoted -v
+go test ./pkg/utils/ -run TestAllYAMLFunctions -v
+go test ./pkg/utils/ -run TestStandardEncodingBreaksAll -v
+go test ./tests/ -run TestTerraformStateWithCustomDelimiters -v
+go test ./tests/ -run TestCustomDelimitersTemplateProcessing -v
+```

internal/exec/describe_stacks.go

@@ -448,7 +448,7 @@ func ExecuteDescribeStacks(
 
 							// Process `Go` templates.
 							if processTemplates {
-								componentSectionStr, err := u.ConvertToYAML(componentSection)
+								componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(componentSection, atmosConfig.Templates.Settings.Delimiters)
 								if err != nil {
 									return nil, err
 								}
@@ -702,7 +702,7 @@ func ExecuteDescribeStacks(
 
 							// Process `Go` templates.
 							if processTemplates {
-								componentSectionStr, err := u.ConvertToYAML(componentSection)
+								componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(componentSection, atmosConfig.Templates.Settings.Delimiters)
 								if err != nil {
 									return nil, err
 								}
@@ -941,7 +941,7 @@ func ExecuteDescribeStacks(
 
 							// Process `Go` templates.
 							if processTemplates {
-								componentSectionStr, err := u.ConvertToYAML(componentSection)
+								componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(componentSection, atmosConfig.Templates.Settings.Delimiters)
 								if err != nil {
 									return nil, err
 								}

internal/exec/terraform_generate_backends.go

@@ -237,7 +237,7 @@ func ExecuteTerraformGenerateBackends(
 				configAndStacksInfo.Stack = stackName
 
 				// Process `Go` templates
-				componentSectionStr, err := u.ConvertToYAML(componentSection)
+				componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(componentSection, atmosConfig.Templates.Settings.Delimiters)
 				if err != nil {
 					return err
 				}

internal/exec/terraform_generate_varfiles.go

@@ -202,7 +202,7 @@ func ExecuteTerraformGenerateVarfiles(
 				componentSection["atmos_manifest"] = stackFileName
 
 				// Process `Go` templates
-				componentSectionStr, err := u.ConvertToYAML(componentSection)
+				componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(componentSection, atmosConfig.Templates.Settings.Delimiters)
 				if err != nil {
 					return err
 				}

internal/exec/utils.go

@@ -722,7 +722,13 @@ func ProcessStacks(
 
 	// Process `Go` templates in Atmos manifest sections.
 	if processTemplates {
-		componentSectionStr, err := u.ConvertToYAML(configAndStacksInfo.ComponentSection)
+		// Use delimiter-safe YAML encoding when custom delimiters are configured.
+		// This prevents YAML's single-quote escaping ('') from breaking template delimiters
+		// that contain single-quote characters (e.g., ["'{{", "}}'"]). See #2052.
+		componentSectionStr, err := u.ConvertToYAMLPreservingDelimiters(
+			configAndStacksInfo.ComponentSection,
+			atmosConfig.Templates.Settings.Delimiters,
+		)
 		if err != nil {
 			return configAndStacksInfo, err
 		}

pkg/utils/yaml_utils.go

@@ -636,6 +636,86 @@ func ConvertToYAML(data any, opts ...YAMLOptions) (string, error) {
 	return buf.String(), nil
 }
 
+// ConvertToYAMLPreservingDelimiters converts data to YAML while ensuring that custom template
+// delimiter characters are preserved literally in the output. When custom delimiters contain
+// single-quote characters (e.g., ["'{{", "}}”"]), the default yaml.v3 encoder may use
+// single-quoted style for certain values (like those starting with '!'), which escapes internal
+// single quotes as ”. This breaks template processing because the delimiter pattern is altered.
+// This function forces double-quoted YAML style for affected scalar values to preserve delimiters.
+func ConvertToYAMLPreservingDelimiters(data any, delimiters []string, opts ...YAMLOptions) (string, error) {
+	defer perf.Track(nil, "utils.ConvertToYAMLPreservingDelimiters")()
+
+	// If no delimiters or delimiters don't contain single quotes, use standard encoding.
+	if !delimiterConflictsWithYAMLQuoting(delimiters) {
+		return ConvertToYAML(data, opts...)
+	}
+
+	// Marshal Go value to yaml.Node tree so we can control quoting styles.
+	var node yaml.Node
+	if err := node.Encode(data); err != nil {
+		return "", err
+	}
+
+	// Walk the node tree and force double-quoted style for scalar values
+	// that contain single quotes (which would conflict with YAML's single-quote escaping).
+	ensureDoubleQuotedForDelimiterSafety(&node)
+
+	// Encode the modified node tree to YAML string.
+	buf := yamlBufferPool.Get().(*bytes.Buffer)
+	buf.Reset()
+
+	defer func() {
+		buf.Reset()
+		yamlBufferPool.Put(buf)
+	}()
+
+	encoder := yaml.NewEncoder(buf)
+
+	indent := DefaultYAMLIndent
+	if len(opts) > 0 && opts[0].Indent > 0 {
+		indent = opts[0].Indent
+	}
+	encoder.SetIndent(indent)
+
+	if err := encoder.Encode(&node); err != nil {
+		return "", err
+	}
+	return buf.String(), nil
+}
+
+// delimiterConflictsWithYAMLQuoting checks if any custom delimiter contains a single-quote
+// character that would conflict with YAML's single-quoted string escaping.
+func delimiterConflictsWithYAMLQuoting(delimiters []string) bool {
+	if len(delimiters) < 2 {
+		return false
+	}
+	return strings.ContainsRune(delimiters[0], '\'') || strings.ContainsRune(delimiters[1], '\'')
+}
+
+// ensureDoubleQuotedForDelimiterSafety recursively walks a yaml.Node tree and forces
+// double-quoted style for scalar nodes whose values contain single-quote characters.
+// This prevents YAML's single-quote escaping (”) from interfering with template
+// delimiters that contain single quotes.
+func ensureDoubleQuotedForDelimiterSafety(node *yaml.Node) {
+	if node == nil {
+		return
+	}
+
+	switch node.Kind {
+	case yaml.ScalarNode:
+		// Only change scalar nodes that contain single quotes.
+		// These are the values that yaml.v3 would single-quote encode, causing '' escaping
+		// that breaks template delimiters containing single quotes.
+		if strings.ContainsRune(node.Value, '\'') {
+			node.Style = yaml.DoubleQuotedStyle
+		}
+	case yaml.DocumentNode, yaml.MappingNode, yaml.SequenceNode:
+		for _, child := range node.Content {
+			ensureDoubleQuotedForDelimiterSafety(child)
+		}
+	}
+}
+
 //nolint:gocognit,revive
 func processCustomTags(atmosConfig *schema.AtmosConfiguration, node *yaml.Node, file string) error {
 	defer perf.Track(atmosConfig, "utils.processCustomTags")()