bug: nico 2.0 site config spiffe_trust_domain not aligned with chart trustDomain; in-cluster services return bare HTTP 403

[kshivakumar-nvidia]

Version

nico 2.0.0-rc.2 (nico-core / nico-api site config)

Describe the bug.

After upgrading to nico 2.0, every in-cluster service client of nico-api
(nico-hardware-health, nico-dhcp, nico-site-agent, GetAllExpectedMachines,
FindMachineIds, etc.) can fail with a bare HTTP 403 and nico-api logs:

Request denied: <addr> FindMachineIds [TrustedCertificate]

The principal is only [TrustedCertificate] — no SpiffeServiceIdentifier.
nico-api trusts the client cert at TLS but refuses to parse its SPIFFE service
identity, so the internal per-method RBAC never grants the call.

Root cause: [auth.trust] spiffe_trust_domain in the site config TOML
(carbide-api-site-config.toml, mounted into nico-api) does not match the trust
domain the service certs are issued under.

At nico 2.0, cert-manager / SPIRE issues identities under
global.spiffe.trustDomain (default nico.local in the nico-core chart).
Migrated sites still carry the carbide-0.8.6 site TOML value
spiffe_trust_domain = "forge.local". The mismatch produces the
nico-hardware-health CrashLoop signature (bare 403 on all service RPCs).

This is distinct from operator carbide-admin-cli 403
(#3662): that defect is
the admin client-cert OU path (OU=Invalid → anonymous). This defect is
in-cluster service SPIFFE identity parsing gated by the site TOML trust domain.

Impact

• Symptom: nico-hardware-health CrashLoop + bare 403 on all service RPCs
  until site config is corrected
• Trigger: carbide → nico 2.0 migration where site TOML still pins
  forge.local while certs are issued under nico.local

Why this should be fixed upstream (not per-cluster override)

Today the trust domain is configured in two independent places that can drift:

Layer	Setting	nico 2.0 value
Helm chart	global.spiffe.trustDomain	nico.local (cert issuer)
Site TOML	[auth.trust] spiffe_trust_domain	often still forge.local (0.8.6 carryover)

Downstream operators must manually keep these in sync via per-site TOML edits. A
fresh nico-2.0 install on a migrated site config 403s out of the box until an
operator patches the TOML and restarts nico-api.

The durable fix belongs in the nico Helm chart / nico-api site-config path:
derive or default [auth.trust] spiffe_trust_domain from
global.spiffe.trustDomain (or inject it at render time) so the site config
cannot silently disagree with the cert issuer. At minimum, nico 2.0 migration
docs/chart hooks should flip the site TOML default when upgrading from carbide.

Workaround

Set spiffe_trust_domain = "nico.local" in the site
carbide-api-site-config.toml overlay, then restart nico-api (it reads config
once at startup; the chart has no configmap checksum, so a config change alone
won't roll the pod):

kubectl -n forge-system rollout restart deploy/nico-api

Expected behavior

On nico 2.0, installing/upgrading the chart should produce a site config whose
spiffe_trust_domain always matches global.spiffe.trustDomain, without
requiring a separate per-cluster TOML edit. Service clients should authenticate
immediately after a carbide→nico migration.

Minimum reproducible example

# 1. Deploy nico 2.0 with global.spiffe.trustDomain: nico.local (chart default)
#    but site TOML still has forge.local (typical migrated site overlay):
kubectl -n forge-system exec deploy/nico-api -- \
  grep spiffe_trust_domain /etc/forge/carbide-api/site/carbide-api-site-config.toml
# -> spiffe_trust_domain = "forge.local"

# 2. Inspect a service client cert SPIFFE URI:
kubectl -n forge-system exec deploy/nico-hardware-health -- \
  openssl x509 -in /var/run/secrets/spiffe.io/tls.crt -noout -text | grep URI
# -> spiffe://nico.local/forge-system/sa/carbide-hardware-health

# 3. hardware-health (and every other service) gets bare 403:
kubectl -n forge-system logs deploy/nico-hardware-health --tail=20
# -> PermissionDenied, grpc-status header missing, mapped from HTTP status code 403

kubectl -n forge-system logs deploy/nico-api --tail=50 | grep 'Request denied'
# -> Request denied: ... FindMachineIds [TrustedCertificate]

# 4. Fix site TOML to nico.local + restart nico-api -> all services recover

Relevant log output

# nico-api
Request denied: <addr> FindMachineIds [TrustedCertificate]

# nico-hardware-health
Could not fetch endpoints: ... PermissionDenied, "grpc-status header missing, mapped from HTTP status code 403"

Other/Misc.

Related operator-auth defect (different path): #3662

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[ajf]

@shayan1995 please take a look