feat: external secrets resolution (provider-neutral; AWS reference) for Connections/Variables

[alissonrosa-lang]

Summary

Support an external secrets backend (starting with AWS Secrets Manager) so Connections and Variables can be resolved directly from the backend, instead of only living in the control-plane vault.

Today

Leoflow stores Variables/Connections in the control plane (encrypted at rest by secretKey, ADR 0019/0021), created via the Airflow-compatible UI (Admin → Variables/Connections) or the API (POST /api/v2/connections). There is no configurable external secrets backend — I checked the reference/helm/docs and found none. The concept only appears as a planning note (project/planning/airflow-connector-compatibility.md, "not yet ADR 0036 ... before any code lands").

Gap vs Airflow

Airflow 3.x ships secrets backends (e.g. SecretsManagerBackend, Vault), so a Connection/Variable can be sourced directly from the external store — single source of truth, no duplication, no bootstrap step.

Use case (real)

A platform that provisions all secrets in AWS Secrets Manager via Terraform (values never in git; ESO delivers the chart secrets). Connections like a Databricks serverless warehouse (used by dbt_group) already live in SM. Today that connection has to be re-created in leoflow's vault (manually in the UI, or a bootstrap that reads SM and POSTs to the API). An external SM backend would let leoflow resolve it from SM directly — no duplication, fully declarative/IaC.

Proposal

• An Airflow-compatible secrets backend chain for Connection/Variable resolution, configurable via leoflow.yaml/Helm (e.g. LEOFLOW_SECRETS_BACKEND + backend_kwargs), starting with AWS Secrets Manager (keyless auth via IRSA/Pod Identity, matching ADR 0035/0056).
• Resolution order: cache → external backend(s) → control-plane vault (parity with Airflow's default chain).
• Feeds ADR 0036.

Benefit

SM as single source of truth; no secret duplication in the vault; IaC-friendly (secret lives only in SM + Terraform state); parity with the Airflow secrets-backend model.

[neochaotic]

Evaluation — makes sense; generalize it; here's the recommended shape

Verdict: worth doing, and it should be generic (not AWS-only), pod-side, ADR-first. Full study in the design notes; summary:

Generic by design — "does it work with other backends?" → yes. A provider-neutral resolver port with per-provider adapters: AWS SM/SSM (IRSA/Pod Identity) as the reference, plus GCP Secret Manager (GKE Workload Identity), Azure Key Vault (Azure Workload Identity), HashiCorp Vault (Kubernetes auth). Anti-AWS-leak pins: authors declare a leoflow conn_id/var (never a provider path); each adapter normalizes value shape; rotation is the provider's job (per-task pods, no reconcile loop). Adding a provider = a new adapter, not a core change.

Recommended two-layer hybrid:

1. ESO / CSI Secret Store — documentation now, zero code (XS). Works today via the shipped taskSecret mount + ADR 0035 key_path: an external store syncs to a K8s Secret, leoflow mounts it. Unblocks the IaC/ESO platform immediately; a permanent escape hatch.
2. A leoflow-native SecretResolver port — the real feature (M, needs a new ADR). Resolves pod-side in the agent, for declared names only, merged into the existing AIRFLOW_VAR_*/AIRFLOW_CONN_* env-export (so it covers operator, @task/python, and bash tasks uniformly — an Airflow-native backend would only cover operators that call BaseHook.get_connection). Control-plane vault stays as the chain fallback. Keyless via the task pod's KSA.

Airflow compatibility is a free bonus (reusing the in-pod Airflow backends as the default adapter), not the driving constraint.

ADR posture: does not need to weaken ADR 0048 — resolving pod-side keeps zero control-plane egress (a control-plane-resolution variant would need a high-blast-radius carve-out and is rejected). It extends ADR 0055 with a definitional note (a declared name may be externally sourced; declaration remains the scope authority) and reinforces ADR 0035/0021. It needs its own ADR (0057 — External secrets resolution) — the "feeds ADR 0036" in the issue is off (0036 is the runtime shim).

Next: a concrete AWS-reference design + a reliability/security review of the resolver (ADR 0048 boundary, ADR 0055 scoping composition, per-provider keyless bootstrap, fail-closed on a required-secret miss) before any code. Not urgent — enterprise track, post-rc.

[neochaotic]

Design ACCEPTED (maintainer) — backlog, tentatively rc2

Resolution is pod-side, keyless, provider-neutral. Summary of the accepted shape:

• The leoflow agent resolves declared conn_id/var names in the task pod just before the task runs — never in the control plane (keeps ADR 0048 intact, zero control-plane egress).
• Chain: declared name → external backend (SM/GCP/Azure/Vault) → leoflow vault (fallback) → env. First hit wins; result becomes AIRFLOW_VAR_*/AIRFLOW_CONN_* so operator/python/bash tasks see it uniformly.
• Keyless via the pod's own workload identity (IRSA/Pod-Identity for AWS; GKE-WI / Azure-WI / Vault-k8s for the rest). The agent holds no credential.
• Provider-neutral port: AWS is the reference adapter; other providers are additional adapters behind the same interface. Config via operator-set secrets_backend + backend_kwargs.
• New ADR 0057 records the decision (the issue's "feeds ADR 0036" was off — 0036 is the runtime shim).

Must-do conditions (from the security review)

• B1 (blocker): relax ADR 0055 D6 registration check (validateDeclaredSecrets, repository.go:1002) to accept a declared name covered by a configured backend without calling the provider — existence proven only pod-side at resolve. The one amendment to an existing invariant.
• B2 (blocker): external branch behind the same liveness/enforce gate (ADR 0055) — a non-live TI resolves nothing externally.
• B6 (major): fail-closed on a required-name miss through the whole chain (composes with Typed DAG-run params with conf validation at trigger time #545/required-params).

Guardrails / scope

• Ship default-off. k3d/kindnet CI stays byte-identical; the real proof is an EKS RC (keyless end-to-end, 3-token coexistence, NetworkPolicy egress to STS/SecretsManager or the Pod-Identity link-local) — CI can't catch these.
• Layer 1 — ESO/CSI is docs-only and available today (existing taskSecret mount + ADR 0035 key_path); can be documented ahead of the native resolver to unblock the IaC/ESO use case immediately.
• UI: no management surface for external secrets (IaC owns them, like Airflow); at most a read-only "sourced externally" indicator — captured as an ADR 0057 sub-item, not in the first slice.

Full design + security review in the design notes (spec/external-secrets-design-study.md, spec/external-secrets-aws-reference-design.md).