docs: reframe cert rotation section as forward-looking design

The certificate rotation section read as if describing bugs in existing
code. Rewritten to present the design decisions prescriptively — how
rotation will be handled, not how to fix what's already there.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: jtschelling

docs/designs/029-grpc-tls-authentication.md

@@ -282,55 +282,52 @@ self-signed Issuer, reusing NVSentinel's existing PKI infrastructure.
 
 ### Certificate Rotation
 
-cert-manager automatically rotates certificates before expiry. The current reference
-implementation loads certificates once at startup — neither the server's TLS cert
-nor the client's CA bundle is reloaded at runtime. This means a cert rotation by
-cert-manager would not take effect until the affected pod restarts.
+cert-manager automatically rotates leaf certificates before expiry. The design needs
+to account for how each component picks up rotated credentials without manual
+intervention.
 
-For SA tokens, rotation is already handled: the client interceptor re-reads the
-token file on every gRPC call, so kubelet's automatic token refresh is picked up
-immediately.
+**SA tokens** are the simplest case: because the client interceptor re-reads the
+token file on every gRPC call, kubelet's automatic token refresh is picked up
+immediately with no additional machinery.
 
-#### Server-Side Cert Rotation
+**TLS certificates** require more consideration. A naive implementation that loads
+certs once at startup would serve stale credentials after cert-manager rotates the
+certificate, requiring a pod restart to pick up the new cert. The approaches below
+avoid that.
 
-The janitor-provider loads its TLS key pair once via `tls.LoadX509KeyPair()` in
-`main.go`. When cert-manager rotates the server certificate, the provider continues
-serving the old cert until restarted.
+#### Server-Side Cert Rotation
 
-**Recommended fix**: Use the `certwatcher` package from `controller-runtime`, which
-watches certificate files for changes and hot-reloads them via the
-`tls.Config.GetCertificate` callback. This is the same pattern already used in the
-janitor for webhook and metrics TLS:
+The janitor-provider should use the `certwatcher` package from `controller-runtime`
+to watch certificate files and hot-reload them via the `tls.Config.GetCertificate`
+callback. This is the same pattern the janitor already uses for webhook and metrics
+TLS:
 
 ```go
-// Existing pattern in janitor/main.go for webhook certs:
 watcher, err := certwatcher.New(certPath, keyPath)
-mgr.Add(watcher)  // starts filesystem watch
+go watcher.Start(ctx)  // starts filesystem watch
 
 tlsConfig := &tls.Config{
     GetCertificate: watcher.GetCertificate,
     MinVersion:     tls.VersionTLS12,
 }
 ```
 
-The janitor-provider does not use `controller-runtime`'s manager, so it would need
-to either:
-
-1. Start the `certwatcher` goroutine manually (it implements `Runnable` with a
-   `Start(ctx)` method)
-2. Use a standalone filesystem watcher (e.g., `fsnotify`) to reload the key pair
+Since the janitor-provider does not use `controller-runtime`'s manager, the
+`certwatcher` goroutine would be started manually (it implements `Runnable` with a
+`Start(ctx)` method). This reuses a well-tested package already in the dependency
+tree.
 
-Option 1 is preferred since it reuses the same well-tested package.
+An alternative is a standalone `fsnotify` watcher, but `certwatcher` is preferred
+for consistency with the janitor's existing pattern.
 
 #### Client-Side CA Bundle Rotation
 
-The janitor controller loads the CA bundle once in `NewCSPProviderDialOptions()` and
-builds a static `x509.CertPool`. If the CA itself rotates (e.g., the self-signed
-Issuer regenerates its CA key), the client will reject the new server cert because
-its cert pool is stale.
+The client needs to trust the CA that signed the server's certificate. If the CA
+itself rotates (e.g., the self-signed Issuer regenerates its CA key), the client
+must pick up the new CA bundle or it will reject the new server cert.
 
-In practice, CA rotation is infrequent — cert-manager rotates leaf certificates
-but the CA key typically remains stable. However, for robustness:
+In practice, CA rotation is infrequent — cert-manager rotates leaf certificates but
+the CA key typically remains stable. Three approaches for handling CA bundle updates:
 
 **Option A — Periodic reload**: A background goroutine re-reads the CA bundle file
 on an interval (e.g., every 5 minutes) and swaps the `RootCAs` cert pool atomically.
@@ -342,8 +339,9 @@ file and reload on change.
 typically planned, accept that the janitor pod needs a restart when the CA changes.
 This is the simplest approach and may be sufficient for most deployments.
 
-The reference implementation currently follows option C implicitly. Options A or B
-can be added if CA rotation without downtime becomes a requirement.
+This design recommends **Option C** as the initial approach, given that CA rotation
+is an infrequent, planned event. Options A or B can be adopted later if zero-downtime
+CA rotation becomes a requirement.
 
 ### Test Compatibility