Persist Workspace mail PKI and authenticate public CA bootstrap

[cassi-volkova]

Decision

Workspace mail PKI is owned by the mail node and persisted on its data disk. It is not generated during the image build and does not require a new Exordos Core certificate API.

The first mail bootstrap creates:

• a Workspace-owned CA key and certificate;
• a leaf key and certificate for the realm-local mail FQDN;
• realm metadata bound to the FQDN and the stable UUID of a dedicated Core password resource;
• an atomic current leaf-generation link.

Subsequent mail root-image updates attach the same data disk, validate the complete PKI state, and reuse the same CA. Backend and mail images may be updated independently after the first coordinated rollout. The migration stage intentionally omits the backend PKI config, so the pinned backend continues its local plain-mail readiness path while the remote mail node initializes independently.

Public CA bootstrap

The current element/config contract is one-way and cannot export a guest-generated file back into a backend config. The mail node therefore provides only the public CA through an internal bootstrap endpoint.

• A dedicated random workspace_mail_ca_bootstrap_secret is delivered separately to mail and backend nodes.
• Each backend request includes a fresh 256-bit nonce and the expected mail hostname.
• HMAC-SHA256 covers the protocol context, nonce, hostname, and exact CA bytes.
• Redirected, replayed, oversized, truncated, modified, or incorrectly signed responses are rejected.
• The backend parses and atomically installs only the public CA before CA-verified SMTP/IMAP STARTTLS readiness can succeed.
• The endpoint runs as a dedicated non-login user with access only to its bootstrap config, public CA, and public realm metadata. It cannot read persistent private keys.

Config-renderer mitigation

The generic config renderer currently creates the destination file before applying its final resource mode and does not publish it through a restrictive temporary file plus atomic replacement. The upstream defect is tracked in gcl_sdk#171.

For the current Workspace rollout, both images install a systemd drop-in for the actual exordos-universal-agent.service with UMask=0077 before the first config delivery. This prevents newly created secret config files from being group/world-readable during the renderer's pre-chmod window.

This is an image-level rollout mitigation, not a generic renderer fix. It must remain in place until the upstream renderer safely creates and atomically publishes config files with their requested modes.

Persistence and validation

Bootstrap must fail closed for:

• a missing or partial persistent PKI store;
• a changed FQDN or realm resource UUID;
• unexpected file types or symlinks;
• unsafe ownership or permissions that cannot be normalized on legitimate regular files and directories;
• a CA key/certificate mismatch;
• invalid or missing CA Basic Constraints or Key Usage;
• a leaf key/certificate mismatch, invalid SAN, invalid chain, or inconsistent combined PEM.

The outer mail-bootstrap lock serializes first initialization and leaf renewal. A leaf inside or beyond the renewal window is replaced by a complete versioned generation signed by the existing CA, then activated through an atomic link update. Corruption is never treated as renewal. Superseded leaf private keys are retained for a bounded rollback window and then pruned.

Backup and rotation

The persistent mail data-disk backup includes CA and leaf private keys. Backup transport, storage, retention, and restore operations must therefore use encryption and signing-key-grade access controls. Restore acceptance includes realm metadata, chain, hostname, key matching, and ownership/mode validation.

CA rotation is an explicit overlapping-trust operation:

1. distribute both old and new public CAs to clients;
2. switch the mail leaf to the new CA;
3. verify all SMTP/IMAP clients and service reloads;
4. retain the old CA for the rollback window;
5. remove the old CA and private key only after convergence.

Deleting the persistent PKI store is not a supported renewal or rotation procedure.

Acceptance criteria

• First initialization creates PKI only after the persistent mail disk is mounted.
• CA and leaf private keys never enter image artifacts, manifests, Core config bodies, backend nodes, API responses, or logs.
• Root-image replacement reuses the persistent CA and valid current leaf.
• Nonce-bound authenticated CA bootstrap and hostname-verified STARTTLS pass.
• Replay, tampering, wrong realm, corrupt state, unexpected types, and partial-state tests fail closed; safe ownership/mode drift is normalized and revalidated.
• Expired-leaf recovery preserves the CA and activates a complete new leaf atomically.
• Old leaf keys are pruned only after the documented rollback window.
• Backup/restore and overlapping CA rotation are validated in an isolated environment.
• The first live introduction updates compatible backend and mail images together without deleting the existing mail data disk.

Current validation status

The implementation is currently a local, uncommitted draft:

• unit and functional suites: 291 passed;
• integration suite: 53 passed;
• focused deployment contract: 39 passed;
• Ruff, shell/Python syntax, and diff whitespace checks pass.

No live deployment or user-facing acceptance test has been performed yet.