feat: support file-based mTLS credentials and TLS server name override

[d2army]

Summary

Two related, additive changes to ConnectionSpec (the renamed TemporalConnectionSpec, post PR #294):

1. Add mutualTLSFilePaths — load mTLS cert/key (and optional CA bundle) from file paths inside the controller pod, alongside the existing mutualTLSSecretRef and apiKeySecretRef modes.
2. Add a top-level TLS server name override (e.g. tlsServerName), applicable to every auth mode that builds a tls.Config, for environments where the dialed hostname doesn't match the cert SAN (PrivateLink, internal endpoints with public-CA-signed certs, etc.).

(1) lets operators feed mTLS material into the controller from any sidecar / init container / CSI driver that writes to a shared volume — e.g. spiffe-helper, cert-manager-csi-driver, or a custom secret-injector — without round-tripping cert/key bytes through a Kubernetes Secret.

Prior context

The existing mutualTLSSecretRef path was recently extended in #158 / #212 to read an optional ca.crt field from the same Kubernetes Secret as tls.crt and tls.key.

That solves the case where all mTLS material can live in one Secret. This proposal is additive on top of that work: it covers environments where the certificate source is file-based (for example, a sidecar or CSI driver writes rotating cert/key material to a shared volume) and where avoiding Secret-backed cert/key material is desirable.

Motivation

Four independent reasons, in priority order:

1. RBAC reduction — no secrets get/list needed for the mTLS path

Today, the controller's ClusterRole includes a kubebuilder RBAC marker that grants:

- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]

…to read the cert/key bytes via fetchClientUsingMTLSSecret. This is a meaningful blast-radius concern in security-sensitive environments because the controller's ServiceAccount needs Secret read permissions broad enough to retrieve any auth material a Connection might reference, not just the specific TLS Secret currently in use.

In file-based mode, the controller reads from disk, never from the API server. The Secret RBAC dependency goes away entirely for clusters that don't also use mutualTLSSecretRef or apiKeySecretRef. (The chart can ship the rule conditionally — out of scope for this issue, see follow-ups.)

2. SPIFFE / SPIRE compatibility

In environments with SPIRE-issued workload identities, the canonical pattern is a sidecar (e.g. spiffe-helper) that writes rotating SVID material to a shared emptyDir volume. The natural fit is a controller that reads that volume directly. Today the only way to bridge SPIRE → mutualTLSSecretRef is to add a custom sidecar that polls the file and writes a kubernetes.io/tls Secret — which is exactly the indirection this issue removes.

Related Temporal ecosystem discussion: in temporalio/temporal#9152, Temporal maintainers noted that the Temporal server already supports hot-reloading TLS roots/certs from disk via RootTLS.RefreshInterval, and were open to SPIFFE-related certificate-provider work in the server. This proposal is narrower: it asks the Worker Controller to support the same practical file-based mTLS delivery pattern, while leaving native SPIFFE Workload API integration as a separate possible future enhancement.

3. Reduced K8s Secret surface area

Some environments restrict controllers from reading Kubernetes Secrets broadly, or require explicit per-Secret approval before storing TLS private keys at rest in etcd. A file-based mode unblocks those orgs without changing the controller's connection semantics — files come in via the pod-spec layer (volumes / CSI), not via the API server.

4. PrivateLink / SNI compatibility (motivates the top-level tlsServerName)

Connecting to Temporal Cloud over PrivateLink — or any environment where the dial endpoint differs from the certificate SAN (e.g. routing through an internal endpoint with a public-CA-issued cert) — requires setting tls.Config.ServerName to the certificate's expected SNI. The current controller API has no field for this, so deployments cannot configure the SDK TLS ServerName through the controller's public API. A top-level field on ConnectionSpec makes this a clean spec-driven knob that applies uniformly to all auth modes that build a tls.Config.

Proposed API

Additive changes to ConnectionSpec (api/v1alpha1/connection_types.go):

// MutualTLSFilePaths configures mTLS using cert/key material loaded from
// file paths inside the controller pod, populated by an init container,
// sidecar (e.g. spiffe-helper), or CSI driver.
type MutualTLSFilePaths struct {
    // CertFile is the path to the PEM-encoded client certificate.
    // +kubebuilder:validation:MinLength=1
    CertFile string `json:"certFile"`

    // KeyFile is the path to the PEM-encoded client private key.
    // +kubebuilder:validation:MinLength=1
    KeyFile string `json:"keyFile"`

    // CAFile is an optional path to a PEM-encoded CA bundle used to verify
    // the Temporal server's certificate. If empty, the system CA pool is
    // used.
    // +optional
    CAFile string `json:"caFile,omitempty"`
}

type ConnectionSpec struct {
    HostPort string `json:"hostPort"`

    // Existing fields unchanged (using the actual upstream types):
    MutualTLSSecretRef *SecretReference         `json:"mutualTLSSecretRef,omitempty"`
    APIKeySecretRef    *corev1.SecretKeySelector `json:"apiKeySecretRef,omitempty"`

    // New: file-based mTLS source
    MutualTLSFilePaths *MutualTLSFilePaths       `json:"mutualTLSFilePaths,omitempty"`

    // New: top-level TLS server name override for SNI / SAN matching.
    // Applies to every auth mode that builds a tls.Config
    // (mutualTLSSecretRef, mutualTLSFilePaths, apiKeySecretRef).
    // Leave empty to default to the host portion of HostPort.
    // +optional
    TLSServerName string `json:"tlsServerName,omitempty"`
}

The exact shape of the SNI knob is up to maintainer preference — happy to land as a top-level tlsServerName string (shown above) or wrapped in a tls: { serverName: ... } TLSOptions struct if you'd rather keep room for future TLS knobs (min version, cipher suites, etc.).

Mutual exclusion (extending the existing CEL rule on ConnectionSpec):

// At most one of mutualTLSSecretRef, mutualTLSFilePaths, or apiKeySecretRef
// may be set.

The deprecated TemporalConnectionSpec (in deprecated_temporalconnection_types.go) is left untouched — new fields land only on Connection.

Implementation sketch

In internal/controller/clientpool/clientpool.go:

1. New auth mode constant:

   AuthModeTLSFile AuthMode = "TLS_FILE"

2. New fetcher function mirroring fetchClientUsingMTLSSecret:

   func (cp *ClientPool) fetchClientUsingMTLSFiles(
       paths v1alpha1.MutualTLSFilePaths,
       opts NewClientOptions,
   ) (*sdkclient.Options, *ClientPoolKey, *ClientAuth, error)

   • os.ReadFile(paths.CertFile) → reuse existing calculateCertificateExpirationTime + isCertificateExpired helpers
   • tls.LoadX509KeyPair(paths.CertFile, paths.KeyFile) instead of tls.X509KeyPair(secret.Data["tls.crt"], secret.Data["tls.key"])
   • Optional paths.CAFile → os.ReadFile then append to system pool, same as the existing secret.Data["ca.crt"] branch

3. Apply TLSServerName in every path that constructs ConnectionOptions.TLS — both mTLS paths (mutualTLSSecretRef, mutualTLSFilePaths) and the API-key path (where TLS: &tls.Config{} is currently initialized empty). The no-credentials path doesn't construct a TLS config and is unaffected.

4. Cache key — the existing ClientPoolKey needs to include any field that affects the resulting tls.Config, including TLSServerName, otherwise a tlsServerName-only change could reuse a stale cached client. Two reasonable shapes — happy to defer to maintainer preference:

   // Option A: add an explicit TLSServerName field to ClientPoolKey, plus a
   //           Source field that identifies the auth material:
   type ClientPoolKey struct {
       HostPort      string
       Namespace     string
       Source        string   // "secret/<name>" for Secret modes,
                              // "files/<sha256(certFile+keyFile+caFile)>" for file mode
       TLSServerName string   // top-level SNI override; empty when not set
       AuthMode      AuthMode
   }
   
   // Option B: collapse everything that affects the tls.Config into one hashed
   //           Source identifier:
   //   "secret/<name>+sni:<tlsServerName>" for Secret modes
   //   "files/<sha256(certFile+keyFile+caFile+tlsServerName)>" for file mode

5. Dispatch in ParseClientSecret (line 246 — probably worth renaming, since it now handles a non-Secret path; e.g. ResolveClientCredentials):

   case AuthModeTLSFile:
       return cp.fetchClientUsingMTLSFiles(*opts.Spec.MutualTLSFilePaths, opts)

   No cp.k8sClient.Get call in this branch — the whole point.

6. Rotation: the existing expiryTime / GetSDKClient flow already triggers a refetch when the cached cert is near expiry. With file-based mode, refetch should re-read the files; if a sidecar (spiffe-helper etc.) overwrites them in place, rotation should flow naturally. Worth covering in tests rather than asserted blindly. A future enhancement could add fsnotify for sub-buffer-time rotation, but is not required for v1.

Backwards compatibility

• ✅ Purely additive. Existing mutualTLSSecretRef users unchanged.
• ✅ CRD fields are optional; defaults are nil/empty; old CRs deserialize identically.
• ✅ Cache eviction logic unchanged (per-key, evicts on expiry).
• ✅ Deprecated TemporalConnection type left untouched.
• ⚠️ One CRD validation rule update on ConnectionSpec (mutual exclusion now includes a third option).

Security considerations

mutualTLSFilePaths is operator-level configuration. A principal who can create or update a Connection could cause the controller to read files from its own pod filesystem. This is similar in shape to the existing mutualTLSSecretRef path, where a principal who can write a Connection can already direct the controller to read whichever Secret is named in that CR — the surface moves from API-server Secrets to pod-local files, not strictly larger.

Two mitigations worth considering:

1. Path allowlist (recommended default) — validate that certFile, keyFile, and caFile are under a configurable prefix (e.g. /etc/temporal/tls/), so a Connection author cannot point the controller at arbitrary files in the controller pod. Happy to implement in the same PR (CEL validation rule, validating webhook, or both).
2. Restrict Connection write access at the cluster RBAC layer — already sensible practice today and unchanged by this proposal.

Helm chart implications (separate follow-up PR)

This is API + controller-code only. To actually use file-based mode end-to-end, the chart needs:

• extraVolumes / extraVolumeMounts to mount the shared cert/key volume into the controller pod
• extraContainers to run spiffe-helper (or whichever cert producer) as a sidecar
• Optional: a values flag that conditionally drops the secrets get/list/watch rules from manager-role when the deployment exclusively uses file-based mTLS

These can land in a follow-up chart PR after the API change merges. Out of scope for this issue.

Open questions for maintainers

1. Naming: mutualTLSFilePaths vs mutualTLSFiles vs mutualTLSFromFiles? Happy to align with whatever convention the team prefers.
2. tlsServerName shape: top-level scalar (shown) vs tls: { serverName: ... } struct? Leaning toward the struct for future extensibility (min version, cipher suites), but happy to follow your call.
3. Cache key shape: any of the options above, or a third you'd prefer.
4. Path allowlist: include in the same PR, follow-up, or skip?

Use case (proposer)

We're evaluating this controller to manage Temporal Cloud workers with Worker Versioning V3. The deployment environment requires SPIRE-issued workload identity for services connecting to external endpoints, so mounting spiffe-helper-managed SVIDs into the controller pod is the natural fit. Avoiding a Secret-reconciler bridge sidecar is what prompted this proposal. We're happy to drive the PR if the proposal looks reasonable.

---

Refs:

• Existing Secret mode (fetchClientUsingMTLSSecret): clientpool.go#L125-L191
• Existing CRD spec (post-PR-CRD rename: TemporalWorkerDeployment → WorkerDeployment, TemporalConnection → Connection #294 rename): connection_types.go
• SPIFFE Helper precedent: github.com/spiffe/spiffe-helper
• Related Temporal server SPIFFE discussion: temporalio/temporal#9152
• Prior controller mTLS CA work: #158 / #212

[d2army]

Scope clarification: controller client first

One nuance worth making explicit: today mutualTLSSecretRef is used in two places:

1. The controller's own Temporal client, via clientpool.go.
2. Worker pod credential projection, via controller-managed env vars (TEMPORAL_TLS_CLIENT_CERT_PATH / TEMPORAL_TLS_CLIENT_KEY_PATH) and a Secret volume mount at /etc/temporal/tls/.

This proposal is intended to start with case (1) only: file-based mTLS for the controller's own Temporal client.

It would not initially change the worker pod credential projection path. Operators using mutualTLSFilePaths would need their worker pods to handle auth separately (e.g., a per-worker spiffe-helper sidecar populating the same paths the worker code already reads from), or we could discuss a follow-up chart/API design for projecting file-based credentials into worker pods as well.

For our use case, worker pods already have their own SPIFFE/mTLS setup, so controller-client-only support is sufficient and keeps the initial change smaller. Happy to expand scope to worker-pod parity in this issue / PR if maintainers prefer that direction.