experiment: Confidential Computing - #13776
Draft
achimnol wants to merge 208 commits into
Draft
Conversation
…nce corpus, Python and WebAssembly bindings
…-before-exec entrypoint, attested time
Confidential attributes on the scaling group across models, REST and GraphQL. Broker client with unreachable and denied as distinct typed errors. The path-scoped authorisation shim, which is also the decision log. Single-use nonce claims against a per-session quota equal to member count. Reference-value store with supersession, bounded coexistence, an attested-identity and pipeline-signature registration gate, and a drain. Release-policy composition under one advisory lock per endpoint, content-addressed and journalled before upload. Session-scoped resource provisioning with a ledger and an orphan reconciler. The measured blob store keyed per image digest. Denial observability. The interim per-image admission belt. The trusted-computing-base grace window with disclosure.
…uest boot gate and agent policy # Conflicts: # .gitignore
…image, attested disk unlock, benchmarks # Conflicts: # .gitignore
… registration, broker compose and guardrails, certificate issuance # Conflicts: # .gitignore # pants.toml
…s, broker client, authorisation shim, reference values, policy composition, session provisioning
…sion netns and whole-GPU passthrough
… seam, namespace plumbing, GPU passthrough, attested readiness gate
Import ExtendedKeyUsageOID from cryptography.x509.oid and build the authority key identifier from the already-narrowed issuer signing key. Read the update rowcount off a CursorResult, matching the repository layer's idiom. Base the confidential REST request models on BaseRequestModel and the response payload on BaseResponseModel, as PathParam, BodyParam and APIResponse.build require. Pass confidential_plane through the openapi generator's build_api_routes call, which was never updated when the confidential tree was added.
…what the guest fetches, launch the guest as the image expects
Trustee resource identifiers are exactly three segments, repository/type/tag, which every path exercised on the rig confirms. The four-segment form would have been refused at provisioning.
Kata 3.31.0 has no kernel-nvidia-gpu-confidential-tarball target. The confidential nvidia rootfs already declares kernel-nvidia-gpu-tarball as a prerequisite, and image.sh finds the kernel by searching the build tree.
The rig has no pip and no ensurepip, and installing them needs apt. A resolver was the wrong dependency for a build whose point is bit-reproducibility; the wheel is now pinned the same way versions.yaml already is.
… too --pax-option is rejected with --format=gnu. Beyond the crash, the staged tree carried build-host uids on our own 68 files, and host-compiled bytecode reached the image from the three trees copied out of the repo. Both move the measurement between build hosts. The strip is scoped to those trees because the upstream rootfs ships its own python bytecode that belongs in the image.
The upstream targets are phony and rebuild the nvidia driver, the guest components and the rootfs on every invocation, so each iteration on our own staging cost a full rebuild. A stamp over the inputs that actually feed it gates the call; BAI_CC_FORCE_UPSTREAM=1 overrides. Also drops three pins nothing read. They implied control over the guest components, trustee and kernel versions that they did not have; those come from the pinned versions.yaml, whose digest is already checked.
…ng for infiniband off A symbol whose dependencies are unmet never reaches .config at all, so requiring an explicit is-not-set line failed three options that were already off. Separately the nvidia-gpu variant builds infiniband as modules for GPUDirect and overrode our request, so the request was wrong rather than unmet.
EXTRA_PKGS installs into the nvidia variant's stage one, but the final rootfs is assembled by cherry-picking binaries out of it, so the network filesystem clients and fuse never reached the image. They are copied in with their library closure resolved from stage one, which the design requires to happen now because adding them later moves the measurement.
…et it SUNRPC_GSS is a hidden symbol; it is selected by RPCSEC_GSS_KRB5 rather than set directly, so asking for it produced a kernel without it. Both lines stay so the verifier confirms the selection took.
Kata 3.31.0 reads the blob from hypervisor.cc_init_data; the agent defaulted to runtime.cc_init_data, which no launch would have found. The shim address the blob is rendered against is pinned to the agent's frozen default, so the two sides cannot drift apart silently.
…ol client into common
…-side mount, zero host storage binds
…h manager-side client key release
…verb off the agent
…lease fetch() ran kbs-client through a pipe into base64, so the shell reported base64's status, which is zero on empty input. A connection refused therefore arrived as "released zero bytes", the same words a policy denial produces, and the one distinction the decision log exists to make was lost at the first link in the chain. The client's status is now taken on its own, and its diagnostics are classified the way broker/kbs.py already classifies them: a denial marker in the client's output means the broker answered and refused, exit 78 with no retry; anything else means the broker could not be reached. Unreachability now retries on the same ladder the credential-broker daemon uses, five attempts with 5, 10, 15 and 20 second backoff, so an outage that ends inside fifty seconds recovers unattended instead of leaving a machine that has already been given up on. Exhausting the ladder exits 75, the temporary-failure class datastore-bootstrap already uses for a service that stayed unreachable, rather than 78. Each verdict is written to /run/backendai-unlock.jsonl, which survives switch-root, and the credential broker reads that file into /var/log/backendai-credentials.jsonl as its first act, so an outage that recovered is still visible as unreachable rows in the decision log rather than being erased by the successful boot that followed it.
The image is debootstrapped with --variant=minbase and an explicit package list, so it carries no D-Bus and therefore no systemd-logind, and nothing else in the tree read the power button. QEMU raised ACPI, the guest ignored it, and the only way to stop a running bundle was to pull its power. That costs an orderly unmount of the encrypted state volume and an orderly PostgreSQL shutdown on every stop of a machine whose entire job is to hold state. A handler is added rather than the defect being documented as intended, because the alternative packages, D-Bus with logind or acpid, would both enlarge the measured image and the running trust base to gain what twenty lines against the power button's input device gain without either. Nothing is conceded to the host by answering: the host can already cut power at any instant, so a graceful path is strictly less capable than what it holds already, and the guest decides for itself what a shutdown does. The build now refuses an image in which the handler is not enabled.
Two fixes to files this image carries, the backup tools' database socket and the credential broker's untrusted-clock clause, were written in /home/devops/bai-cc against the stale copies of control-plane/ and credential-broker/ that were left in that tree's working directory when the bundle work moved to its own branch, committed to cc/deploy-drain-scoping, and never reached the branch that builds the image. Nothing anywhere said so; the image was simply older than the tree the fixes lived in, and the only way it surfaced was reading the exception tuples back out of the measured artifact. The build now refuses to run against uncommitted bundle sources, so an image always names a revision, and records the branch, the commit and every commit on any other local branch that touches control-plane/ or credential-broker/ and is not in this image, by patch equivalence, into reference-values.json. The reference values are what gets registered, so a divergence that used to be invisible is now carried in the record that describes the artifact.
Every tool the confidential storage bring-up runs is launched as "ld-linux-x86-64.so.2 --library-path ... <tool> <args>", because the staged noble toolchain cannot use the kernel container's own interpreter. gocryptfs backgrounds itself by re-executing os.Executable(), which reads /proc/self/exe, and under that wrapper /proc/self/exe is the dynamic loader. The child was therefore the loader handed "-fg" as its program, and the mount died with "-fg: error while loading shared libraries" before any folder key was used. The re-exec cannot be repaired from the outside: the loader is what the kernel executed, so no shim on PATH and no argv[0] rewrite changes what /proc/self/exe names. A folder whose descriptor names the legacy format now runs gocryptfs with -fg, so nothing re-executes, and the bring-up owns the backgrounding itself: attach() starts the reader detached, feeds it the folder key, and waits for the mount to appear or for the reader to die, reporting whatever the reader complained about. The current format keeps the launcher it had, because its driver backgrounds itself correctly.
The confidential storage driver replied ok to every FUSE fsync without touching the backing file, and it requests FUSE_WRITEBACK_CACHE, so the guest kernel buffered dirty pages and the only thing that ever made ciphertext durable was the backing filesystem's writeback timer. O_SYNC and O_DSYNC are turned by the guest kernel into an fsync after the write, so dropping the fsync dropped the flags with them: a write-ahead log on an encrypted folder believed it was on disk when it was not. Traced on the rig, twenty O_SYNC records produced twenty-three pwrites and zero fsyncs. fsync now syncs the file the handle holds, and its datasync argument picks between fsync and fdatasync rather than being discarded. statfs passed the backing mount's numbers through unchanged, so a folder on an NFS export advertised that mount's one-mebibyte block size while the frame stride is 65576 bytes and the plaintext chunk is 65536. It now reports the chunk as the block size, which is the granularity a caller should write at, and counts whole frames, so the advertised capacity is the plaintext a tenant can actually store rather than the raw bytes the ciphertext will consume.
The confidential agent can create the host sandbox from a stand-in image instead of the session's own, because Kata's ShareRootFilesystem returns forceGuestPull before it ever reads the host rootfs and the guest pulls the reference in its io.kubernetes.cri.image-name annotation. What the host image was still supplying was the OCI process configuration: nerdctl reads Env, WorkingDir and User out of the local config blob, so under a stand-in the container came up with the stand-in's own and no error anywhere. On cc/vllm that silently drops LD_LIBRARY_PATH, the CUDA PATH entries and the NVIDIA_* variables. The registry scan already fetches the image config blob and keeps only its Labels. It now keeps Env, WorkingDir and User too, in a new images process_config column that rides to the agent inside ImageConfig, and the confidential agent seeds the container environment from it before overlaying the manager's own variables, exactly the precedence a local image would have given. Cmd and Entrypoint need no carrying: the agent always passes a non-empty kernel-runner argv under an explicit entrypoint, so the image's own were never consulted. A row that predates the column carries an empty mapping, which is indistinguishable from a lossless start unless someone looks. With a stand-in configured the agent refuses such a kernel rather than booting it stripped.
Pointing them at the bundle's own socket got them as far as the server and no further. PostgreSQL is initialised with --auth-local=scram-sha-256, so a local connection is authenticated like any other, and pg_dump ran as root with nothing to present: "connection to server on socket /run/backendai/.s.PGSQL.5432 failed: fe_sendauth: no password supplied". The hourly timer had therefore still never produced a segment, and the earlier PGHOST fix, which was necessary, was read as sufficient because nobody had run the unit since. backendai-state-backup.service now asks the credential broker for the datastore password as well, the release policy grants that pair, credential-guard refuses a boot in which it arrives zero length like every other credential the unit takes, and both tools export PGUSER and PGPASSWORD alongside PGHOST.
The deployment and drain-scoping line met the confidential-computing line where the guest storage helper, the coco agent's image spec and the image model overlap. Nothing collided textually: git resolved every file on its own, and the three-way result was checked by hand rather than trusted. Two of the four commits carried no new content. The storage-fuse commit that honours a guest fsync and reports the encrypted geometry is a patch-identical duplicate of 16ec570, already on this branch, so the merge leaves rust/bai-storage-fuse untouched. The commit pointing the backup tools at the bundle's own database socket is patch-identical to the same fix on cc/control-plane-cvm; it lands here first, and the later merge of that branch resolves the overlap in favour of the fuller version that also presents a datastore password. The new migration b8e2d4f60a91 for the image process-configuration column descends from f31c6a8e04b7 and is the single head of the revision graph; no existing revision was rewritten in place, so a fresh database and an already-migrated one reach the same shape. The release-policy renderer is untouched by this merge: the guard that moves reference values leaving a required measurement unpinned into an invalid state, and list-valued rtmr_2 rendered as Rego set membership, both survive verbatim.
The control-plane trust-domain line met the confidential-computing line in the state bundle's backup tools, the initramfs unlock path, the bundle build and the credential broker. That branch forked as a worktree and was never merged back, and the work it carries began life as uncommitted edits in the main checkout, so both sides hold near-duplicates of the same fixes; the resolution below sorts genuine content from repetition rather than preferring a side. Two of the six commits carried no new content. The credential broker's untrusted-clock clause is textually identical to de75657, already on this branch, and merged to nothing. Pointing the backup tools at the bundle's own database socket is patch-identical to the same fix carried on cc/deploy-drain-scoping and merged one commit earlier. That last duplicate is the only collision. Both sides set PGHOST to the bundle's own socket directory, which merged cleanly; they then part over the export. This side exported PGHOST alone, the incoming side exports PGHOST, PGUSER and PGPASSWORD and reads the datastore password from the unit's credential directory. The incoming side is a strict superset and is taken whole in both state-backup and state-restore. It has to be: the emptiness guard on PGPASSWORD merged in cleanly outside the conflicted region, and under set -eu the shorter export would leave that guard dereferencing an unset variable and abort every backup at line eighteen. Everything else merged without collision. The power-button handler, the build's refusal to measure uncommitted sources, the initramfs telling an unreachable broker apart from a refused release, and the broker carrying initramfs verdicts into its decision log all arrive intact, and no file under src/ or rust/ changes. The release-policy renderer keeps both of its semantics: reference values leaving a required measurement unpinned are moved to an invalid state and contribute no rule, and a list-valued rtmr_2 renders as Rego set membership. Rendered offline against the live reference values, the composed document for the manager's broker is byte-for-byte the document that broker is serving.
…ngs to An ocicrypt-encrypted image names its layer key encryption key by a broker resource reference frozen into the manifest at encryption time. The reference carries no session term and never can, so the broker's release predicate, which sees only the attestation body and the resource path, would hand that key to any attested guest at any time for as long as the image exists. The authorisation shim now recognises a three-segment path under the image-key segment and releases it only while this manager is launching or running a session in that domain on the image whose canonical reference hashes to the tag. The tag is the sha256 of the canonical reference, so no schema, no migration and no registration surface is needed: an image without a key in the broker never has one asked for, which is what makes the property elective and per image. An unattested fetch and a fetch for an image no live session runs are refused with typed errors and a decision row. This narrows, it does not exclude. An operator can start a session on the image and be released the key legitimately.
One commit is new on this branch since it was last merged here. The layer key encryption key of an ocicrypt-encrypted image is now released only while this manager is launching or running a session, in that domain, on the image whose canonical reference hashes to the tag in the resource path. An unattested fetch, and a fetch for an image no live session runs, are refused with typed errors and a decision row. This narrows the release predicate; it does not exclude an operator who legitimately starts a session on the image. The single conflict was the import block of confidential/shim.py. This branch adds the SessionRow import the new predicate needs, and experiment/confidential-computing had added execute_with_txn_retry alongside a module logger on the adjacent lines. Resolved as the union of both sides, with no statement dropped from either. All 92 lines the commit adds are present verbatim in the merged tree. The release-policy renderer is untouched by this merge. Rendering the document offline from this tree against the live reference values at endpoint 18088 reproduces the deployed document at kbs-dev-state/storage/kbs/resource-policy.rego byte for byte, sha256 1b27829271215044ca749c406b14c68805fa9df975a40bffd8f44f1c779c0608, which is the same document the pre-merge baseline rendered and the same one on disk before the merge started. The merge changes nothing the broker enforces. opa check passes on the rendered document in its own directory.
A scaling group that names a tenant launch authority no longer lets the manager mint its own launch nonce. LaunchAuthority.mint takes the oldest unspent credential whose endpoint, domain, image digest and member count match the session being provisioned, marks it spent under a row lock with skip_locked, and binds it to that session id. With no authority configured the previous secrets.token_urlsafe path is kept verbatim, so an unconfigured deployment behaves as before. A credential is deposited through POST /confidential/launch-credentials under the admin middleware. The deposit verifies an Ed25519 signature over a context-prefixed canonical JSON statement covering the nonce, the domain, the image digest and the quota, so a credential the authority did not sign is refused with LaunchCredentialRefused before it reaches the database, and the nonce primary key makes a redeposit of a spent credential a refusal rather than a restoration. Layer key encryption key release now consumes a launch-nonce claim slot. The shim previously found the first launching session on the image and returned it; it now collects every launching session together with its nonce and consumes a claim against each in turn, returning the first that had a slot left. A guest that is measurement identical to the entitled one but is not the session that spent the credential therefore competes for a finite quota rather than being served silently, and the release is recorded in the decision log against the nonce it consumed rather than against nothing.
… token The measured configuration document now carries an image security policy inline in cdh.toml. Its default is reject and the only admissible scope is the deployment registry, requiring a cosign signature over the image manifest digest against a public key written into the document itself, so no key fetch crosses the wire and swapping either the policy or the key moves mr_config_id. Until this key existed image-rs built no signature validator at all, logged that all images can be pulled without filtering, and pulled whatever it was handed. backendai.toml pins the key broker's attestation token signing key, and bai-guest-boot now verifies the ES256 signature over the token it anchors its clock on before reading any claim out of it. The verification is a P-256 check written out here because the guest carries no asymmetric crypto library; the pinned key is the subject public key info of the broker's own signer, taken from the measured document rather than from the token header, so a substitute broker in the path produces a token the guest refuses. The fallback that read the issue time out of an ordinary broker resource is removed with it: a fallback a substitute broker can also serve turns the check into one an adversary declines by answering differently. The guest starves on refusal rather than proceeding. The configuration bundle, the mount plan and the channel identity are all fetched by bai_cc_bootstrap only after the ready file this path writes, so a guest that cannot authenticate its broker consumes nothing from it.
The decision log existed so that an operator could tell an authorisation refusal apart from an infrastructure outage after the fact, and it was written to /var/log, which var-log.mount makes a tmpfs. The record was therefore destroyed by the stop of the machine that wrote it, which is the one moment it is wanted. It now lives under /var/lib/backendai, the attested state volume, which the initial ramdisk unlocks before any unit starts and which the credential broker now requires, so the file the initial ramdisk's verdicts are carried into outlives the trust domain. The destination is checked rather than assumed. The log constructor refuses a path outside the declared durable root and refuses a durable root that is not a mount point, so a boot in which the state volume is absent fails loudly instead of appending to the read-only root, and the build refuses an image whose policy names a destination off the volume.
The datastore benchmark called psql with no host, user or password and the coordination-store benchmark called etcdctl with no authentication, so both measured a count of zero against a bundle whose PostgreSQL listens on /run/backendai and whose etcd requires a root password. Both now take the same route the backup tools take: the harness reads them from its unit's credentials directory and refuses if either is absent, and a failed iteration keeps its first error rather than reporting an unexplained zero. The harness had no unit and was not installed in the image, and restore had no unit either, so neither could be handed credentials at all. Both now have one, with the pairs they need declared in the policy table.
…heckers IdleCheckerHost.update_app_streaming_status iterated only self._checkers, and the only implementation that does anything with an app-streaming status lives on NetworkTimeoutEventDispatcherIdleChecker, which the host keeps in self._event_dispatch_checkers. The agent metered every confidential session channel circuit, anycast SessionChannelActivityAnycastEvent every thirty seconds, and the manager consumed the event and threw the figure away, so an open channel never suspended the idle timer and a closed one never refreshed it. The host now dispatches to both lists and the abstract event-dispatcher checker carries a no-op default so an implementation that does not care is unaffected.
The unit declared Requires=backendai-credentials.service, and one of the five benchmarks it runs restarts that broker to time the credential fan-out. A required unit stopping stops the units that require it, so the harness was sent a termination signal in the middle of its own fourth benchmark and reported nothing at all. Ordering after the broker is what the unit actually needs; the credential load fails loudly on its own if the socket is absent. The default argument line was also unquoted, so the service manager read its second word as a second assignment and said so at every start.
Every other unit that loads a brokered configuration file names its own working directory, because the credentials directory it reads from is read-only and mode 0500 and anything writing beside its configuration file would fail there. The event bus was the exception: it loads a rendered valkey.conf and inherited the root directory, which under its own ProtectSystem=strict is read-only. Nothing has failed because the rendered configuration names an absolute directory of its own, so the gap was latent rather than live, and it is closed here so the property holds by construction rather than by a second mechanism.
The control-plane worktree forked at the commit that gave the backup tools a datastore password and was never merged back, so four fixes to the state bundle lived on a branch the measured image never reads. They are the durable decision log on the encrypted state volume, credentials and a unit for the benchmark harness and for state restore, the benchmark unit no longer requiring the broker it restarts mid-run, and an explicit working directory for the event bus. Nothing on the feature branch touched control-plane or credential-broker after the fork, so every one of the ten files merged without a collision and each is byte-identical to the branch tip. The four commits are all new work; none duplicates anything already carried here. The commit giving the event bus its working directory is ahead of the artifact that was built and measured, and rides the next bundle rebuild.
Three commits met the feature branch: tenant-signed launch credentials, the guest-image signature policy together with the pinned broker token signer, and the idle-checker dispatch fix. Nothing conflicted textually, but three files carried work written on both sides against different context and had to be checked rather than trusted. In the authorisation shim the branch turned _launching_image_session into _launching_image_sessions and made a layer key encryption key release consume a launch-nonce claim slot, while this branch had independently rewritten _consume to run under execute_with_txn_retry and to backfill the reference value the attested guest was admitted by. The merge keeps this branch's _consume, which is the newer of the two, and the branch's new callers on top of it. The two are signature compatible: _consume returns the session id and whether a slot was newly taken, and the new image-key loop takes the same tuple and raises the same NonceQuotaExhausted, which is what it does on its own branch too. That loop discards the newly-taken flag, so a claim taken for an image key and then abandoned because the broker was unreachable waits for its lease to expire rather than being released at once; that is the branch's own behaviour and is carried across unchanged rather than repaired here. In provisioning the branch replaced the manager's own secrets.token_urlsafe mint with LaunchAuthority.mint while this branch had reworked the same function to survive a serialization pivot. The mint replaces only the nonce expression and stays outside the retried settlement, so both survive. The nonce row, the scaling group options and the REST handler each gained additive members on the branch that no work here touches. The release policy renderer is untouched by this merge. The guard that moves a reference value pinning less than the required measurements into a state that admission excludes, and the rendering of a list-valued rtmr_2 as set membership where a non-empty list counts as pinned and an empty one does not, both live in policy.py and references.py and are unchanged. The migration tree keeps a single head. The branch added one revision and rewrote none in place; it chains onto the revision this branch already carried.
… onto the branch The deployed tree sat sixty-five commits behind experiment/confidential-computing with sixty-one uncommitted files, and the working assumption was that the lag was one-sided, so the tree could simply be brought forward. It was not. Seven of those files held work committed nowhere, and for six of them the branch blob was byte-identical to the deployed blob, meaning the branch had never seen the change at all rather than having superseded it. It presented as nothing at all, which is why it nearly went. The checkout reports success and the manager starts. The loss would have surfaced later and elsewhere: folder-key releases leaving no decision row, a clone of an encrypted folder coming back untiered, and purge leaving a live key in the broker. DecisionActor.CLIENT is the sharpest case, because it is a regression and not a missing feature. The deployed database already holds ninety-nine decision rows whose actor is client, and ConfidentialDecisionRow coerces that column through StrEnumType(DecisionActor), so dropping the member turns every one of those rows into a load failure the next time the decisions are listed. Restored here: the CLIENT actor and the journalling in client_keys that writes it, with the shim wired through plane; the encryption tier carried through VFolderCloneInfo, the repository read and clone paths and the service layer, so a clone inherits its source tier and its key; folder-key revocation on purge and force-delete, which had reached one call site out of four; and refuse_immutable_updates, dispatched generically from the base updater to close the bulk-update path that the ORM attribute listener never sees, because a Core update never fires the set event. Discarded deliberately: the in-place rewrite of migration c1f0a7d3b9e2, which the branch already carries forward as d5b8f2c47a10. Holding both would have a fresh database create confidential_nonces without claims_used and then drop a column that had never existed.
The branch was written on 2026-08-05 and then left unmerged while every other topic branch of the campaign landed, so the only copy of it was a rig-local ref and a remote branch nobody was tracking. Nothing about it was contentious; it simply outlived the wave that wrote it. It removes the opt-in integrity tier. The tier never had a working guest half, and what remained was a manager and storage skeleton with no call sites and no client surface. It was also making a claim it could not keep: encryptionTier was accepted unvalidated, nothing branched on it, and client key release read it back to tell the client its folder was tamper-evident. A folder created with that flag got ordinary concurrent-tier storage while its owner was told otherwise, and any other string raised KeyError at the same site. Two files had moved on this side since the fork and both auto-merged. cc_storage.py kept the tier lookup falling back to the concurrent disclosure rather than raising. client_keys.py is the one worth naming, because this side had just gained the DecisionActor.CLIENT journalling that had itself nearly been lost; the merge keeps both, so folder-key releases are still journalled and the tier is still gone. The migration deletion was the risk and it was checked rather than assumed. The branch deletes d3e1b7a90c4f outright, on the grounds that integrity_mount_leases was never created in any live database, and re-points d2e4a6c80b57 at c1f0a7d3b9e2. After the merge the graph holds 413 revisions with a single head at a4c7e1b09d33 and no down_revision referencing an absent file.
This merge changes no file. It is recorded anyway, because the branch's nine commits are otherwise unreachable and their messages are the only account of why each change was made. The content arrived by a different route and the duplication is worth naming. The same work sat uncommitted in the rig's working tree, sixty-five commits behind the branch, and was rescued from there earlier today as a single commit after an audit found the feature branch did not carry it. That audit compared the deployed blobs against the feature branch and was right about both. What it did not do was look for the work on a topic branch in the workstation clone, where it had been properly committed on 2026-08-06 and simply never merged. Four of the seven rescued files are byte-identical to this branch. So the commit graph reported nine commits of divergence while the trees agreed. That is the mirror of the trap the rig notes already record, that patch-id comparison under-reports duplicates across this divergence and blob hashes must be compared too: here the graph over-reported it. Merging and finding an empty result is what settled it, and it is cheaper than reading nine diffs. Everything the branch adds is present: destroying a folder key on every deletion path, the audit row for a client folder-key decision, keeping a lapsed window open under a running guest, carrying the encryption tier through a clone and through the accessible-folder path, refusing a tier change where the update is actually written, and leaving session sweeps to destroy their own resources.
…ectory cc/ subdirs
The branch had grown six root-level directories (guest-image, control-plane,
credential-broker, trustee, host-setup, rust) for what is a feature spanning the
existing components rather than a component of its own. Fold the auxiliary
material into the semantic directory that already owns its kind:
scripts/cc/{guest-image,control-plane,host-setup,trustee} build & provisioning
configs/cc/{guest-image,control-plane,credential-broker,trustee} pins, units, policies
docker/cc/ builder and PKIX dockerfiles, Trustee compose stack
Payload keeps its path contracts: guest-image/overlay stays one tree beside the
rootfs.sh that cp -a's it, control-plane/bin stays beside the build script that
glob-installs it, and the dracut module keeps module-setup.sh next to the script
it inst_script's. Build scripts gained an explicit config root (BAI_CC_CONFIG,
CONFIG) now that configuration lives in configs/cc.
Promote the credential broker to a regular component at src/ai/backend/cc_broker
with full BUILD configuration: it joins the pants build, so its imports are
absolute, its signatures are annotated, and it is lint-clean. It stays
stdlib-only and runs on the image's bare python3 as before, which keeps the
measured TCB unchanged.
rust/ stays at the root as a semantic top-level directory of its own.
Claude-Session: https://claude.ai/code/session_01MYyrHQVMozn88PQDNM4mNT
achimnol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds confidential computing to Backend.AI: compute sessions run inside hardware-attested confidential VMs, the control plane itself runs as a measured state bundle, and vfolder content is encrypted under keys released only against a satisfied attestation policy.
Confidential computing is a feature spanning the existing components, not a component of its own. Implementation lives beside the code it extends under
src/ai/backend/; the auxiliary build, provisioning, and deployment material is gathered intocc/subdirectories of the matching top-level directory.282 files changed, +21627 / −1013, across 207 commits.
What's in it
Manager
manager/confidential/manager/api/rest/confidential/manager/models/confidential/manager/errors/confidential.py,manager/metrics/confidential.py10 alembic migrations: the confidential computing plane, tenant launch credentials, tunnel port, channel vouching, attested guest witness, nonce-claim lease, client-key release and folder tier, image process config, session→reference-value attribution, and appproxy worker upstream TLS capability.
Agent and guest
agent/coco/— confidential-container backend (kernel, runtime, netns, relay, volumes, hostlock, blob, resources, spec)kernel/guestops.py,kernel/terminator.py— guest-side operationscommon/kernel_runner/— the guest channel protocol, transport, and vocabularycommon/pki/— issuance, chain, and types shared across the manager and guestStorage format
rust/holds five crates implementingbackend.ai/cc-storage/v1— the format, a frozen conformance corpus, Python and WASM bindings, and a FUSE layer.rust/bai-storage-format/FORMAT.mdis normative for the bytes; the corpus decides where it and an implementation disagree.common/cc_storage.pybinds it into the Python side.Measured control plane
src/ai/backend/cc_broker/is a regular component: an attestation-gated credential broker that runs inside the measured state bundle, releasing credentials only against a satisfied policy and recording every verdict to a durable decision log. It is deliberately stdlib-only and runs on the image's barepython3with no venv, which keeps the measured TCB small.Auxiliary trees
scripts/cc/guest-image/overlay/tree copied verbatim into the guest filesystemscripts/cc/control-plane/bin/,bench/), dracutinitramfs/modulescripts/cc/host-setup/scripts/cc/trustee/configs/cc/*docker/cc/Payload keeps its path contracts:
guest-image/overlay/stays one tree beside therootfs.shthatcp -a's it,control-plane/bin/stays beside the script that glob-installs it into/usr/lib/backendai/, and the dracut module keepsmodule-setup.shnext to the script itinst_scripts.README.mdgains a### Confidential Computingsection mapping all of the above. The existing Directory Structure list is deliberately untouched — this is a feature, not a top-level component.Reproducibility
The state bundle is built in a pinned container and attributed to a revision:
build-in-container.shrefuses to build from uncommitted bundle sources unlessBAI_ALLOW_UNCOMMITTEDis set, and warns when other branches carry bundle commits the image will not. The introspection console is compiled out of the measured image and the build asserts no implementation survived.Status — what is not verified
This is the honest list; please weigh it before reviewing depth-first.
proposals/per the repo's own BEP-first rule. None exists yet. This is the largest documentation gap.README.md/AGENTS.md/CONTEXTS.md. Notably, design rationale currently lives inechostrings inside build scripts (e.g. why the introspection console is compiled out, why RTMR values must come from a booted TD) rather than in prose.pants check(mypy) not run locally — it requires CPython 3.13.7, absent from the dev machine used here. This affects the whole branch, not just recent work. Needs CI.agent/coco/agent.py,manager/confidential/launch.py,sokovan/scheduler/*,dependencies/orchestration/*,common/dto/manager/session/types.py.tests/unit/agent/test_legacy_inference_env.pyimportsai.backend.agent.legacy_inference_env, which does not exist on this branch.pants checkfails withUnownedDependencyErroras a result.make -C scripts/cc/guest-imageandbuild-in-container.shneed a real TDX host, and the recent config-root rewiring (BAI_CC_CONFIG,CONFIG,BROKER_CONFIG) has not been run end-to-end since.Related
Incorporates #13775 (the
cc/layout refactor andcc_brokerpromotion), merged into this branch.https://claude.ai/code/session_01MYyrHQVMozn88PQDNM4mNT