[security] Workspace SSH host private keys are persisted in cloud provider user-data

[coygeek]

[security] Workspace SSH host private keys are persisted in cloud provider user-data

Summary

Workspace provisioning generates an Ed25519 SSH host key pair so the coordinator can verify the SSH server it connects to for workspace terminal and native VNC access. The private half of that generated host identity is then copied into the lease config and rendered directly into Linux cloud-init ssh_keys user-data. AWS sends that blob as EC2 UserData, Azure sends it as VM customData, and GCP sends it as instance metadata user-data.

This crosses Crabbox's supported security boundary because the maintainer security policy treats "generated credentials being exposed beyond the current OS user by Crabbox" as in scope. Repository-selected commands and trusted project automation are expected behavior, but this is a Crabbox-generated credential for a coordinator-managed workspace identity. Persisting it in provider metadata exposes the credential to the provider account/control plane and any IAM principal or audit-log reader that can inspect launch user-data for the deployment.

Affected Components

• worker/src/fleet.ts:3353-3359: Workspace provisioning generates a per-workspace Ed25519 SSH host key pair and immediately uses it to build the internal lease creation request.
• worker/src/fleet.ts:13293-13309: workspaceLeaseRequest base64-encodes both the generated private and public host keys into internal request headers.
• worker/src/fleet.ts:2041-2057: The lease creation path requires those workspace host-key headers and copies the decoded private key into LeaseConfig.sshHostPrivateKey.
• worker/src/bootstrap.ts:55-73: Linux cloudInit includes the optional SSH host-key block in the top-level cloud-init document.
• worker/src/bootstrap.ts:688-701: optionalSSHHostKeys renders config.sshHostPrivateKey verbatim under ssh_keys.ed25519_private.
• worker/src/aws.ts:1155-1163: AWS instance creation sends the rendered bootstrap data as EC2 UserData.
• worker/src/azure.ts:1142-1185: Azure VM creation base64-encodes the same Linux cloudInit(config) output and attaches it to the VM osProfile.
• worker/src/azure.ts:1246-1257: Linux Azure VMs place that value in osProfile.customData.
• worker/src/gcp.ts:293-303: GCP VM creation places the same cloudInit(config) output in instance metadata item user-data.
• worker/src/fleet.ts:13671-13693: Workspace SSH connections verify the observed server host-key fingerprint against the stored expected fingerprint, so disclosure of the matching private key undermines an active host identity check.

Attack Path

An operator provisions a coordinator-managed workspace on a cloud provider-backed target such as AWS, Azure, or GCP. During provisioning, the coordinator generates a fresh Ed25519 SSH host key pair for that workspace and stores the public-key fingerprint for later SSH host verification. The private key is then carried through the internal workspace lease request into LeaseConfig.sshHostPrivateKey.

When the provider adapter builds the VM bootstrap payload, Linux cloud-init receives an ssh_keys section containing the full private key. The provider adapter then sends that bootstrap payload to the cloud provider as launch user-data, Azure custom data, or GCP instance metadata.

An attacker does not need Crabbox application access for this step. Any principal with sufficient read access in the cloud account, console, provider support path, or audit-log storage for instance creation/user-data can retrieve the generated SSH host private key after the workspace is launched. With that key and a suitable network position, the attacker can impersonate the workspace SSH server and satisfy Crabbox's expected host-key fingerprint check for subsequent workspace SSH, terminal, or native VNC connections.

Impact

The exposed value is a generated workspace credential, not trusted repository automation or an operator-supplied project secret. Disclosure gives cloud-account readers the private key for the SSH server identity that Crabbox expects for that workspace.

The practical impact is bounded by the workspace lifetime and by the attacker's need for provider-side read access plus a path to intercept or redirect later SSH traffic. Within those bounds, the leaked key can defeat Crabbox's host-key verification for the affected workspace, enabling SSH server impersonation and confidentiality/integrity compromise of later workspace SSH-backed interactions.

Severity Assessment

CVSS Assessment

Metric	v3.1	v4.0
Score	4.3 / 10.0	5.3 / 10.0
Severity	Medium	Medium
Vector	CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N	CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:N/VC:L/VI:N/VA:N/SC:N/SI:N/SA:N
Calculator	CVSS v3.1 Calculator	CVSS v4.0 Calculator

The score reflects a provider-authenticated attacker who can read deployment user-data or equivalent audit material. The direct vulnerable-system impact is confidentiality loss of a generated workspace host credential; the downstream SSH impersonation risk depends on additional network positioning, and the per-workspace key limits blast radius.

Recommended Remediation Approach

Avoid placing generated SSH host private keys in provider-controlled user-data or metadata. The safer design is to generate the host key inside the guest during first boot and return only the public host identity to the coordinator through a narrowly scoped, authenticated readiness channel. If coordinator-side generation remains necessary, deliver the private material over a channel that is not retained as provider launch metadata and document the trust properties clearly.

Add regression coverage that fails if SSH private key material appears in Linux cloud-init output, AWS EC2 user-data, Azure custom data, or GCP instance metadata. Existing workspace host-key tests should be updated to verify that only public host identity or fingerprints leave the guest/trusted channel.

Validation

Validation used source review and targeted test inspection against commit 52d272fbc45e49a6cf2a8027a996379eeadcd0c7.

• worker/src/fleet.ts:3353-3359 generates the workspace host key pair, and worker/src/fleet.ts:13293-13309 places the private key in an internal lease request header.
• worker/src/fleet.ts:2041-2057 decodes that header and stores the private key on the lease config used by provider provisioning.
• worker/src/bootstrap.ts:55-73 and worker/src/bootstrap.ts:688-701 render the private key into the Linux cloud-init ssh_keys.ed25519_private field.
• worker/src/aws.ts:1155-1163, worker/src/azure.ts:1142-1185, worker/src/azure.ts:1246-1257, and worker/src/gcp.ts:293-303 show that the same bootstrap payload is sent to AWS, Azure, and GCP provider launch metadata paths.
• worker/src/fleet.ts:13671-13693 confirms Crabbox later verifies SSH host identity against the generated public-key fingerprint, so disclosure of the corresponding private key affects a real security check.
• worker/test/bootstrap.test.ts:65-77 currently asserts that cloudInit() contains ed25519_private private-key material when sshHostPrivateKey is set, and worker/test/fleet.test.ts:5530-5617 asserts workspace lease creation supplies a generated OpenSSH private key to the provider config. No inspected regression test asserts that generated SSH host private keys are excluded from provider user-data or metadata.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 7, 2026, 4:43 AM ET / 08:43 UTC.

Summary
Keep open: current main still carries a generated workspace SSH host private key into Linux cloud-init and then into provider bootstrap metadata, so the reported security boundary issue remains live.

Reproducibility: yes. for source reproduction: current main clearly generates the workspace host key, renders the private half into cloud-init, and sends that bootstrap through provider metadata paths. I did not perform a live provider launch.

Next step
Security-sensitive coordinator/provider trust-boundary work needs maintainer direction before a repair PR should be queued.

Maintainer decision needed

• Question: Should workspace host identity be generated in the guest with an authenticated public-key readiness handoff, or should Crabbox keep coordinator-side generation but deliver the private key through a non-retained channel?
• Rationale: The unsafe dataflow is clear, but the fix changes the coordinator/provider trust boundary, workspace readiness protocol, and host-key pinning behavior.
• Likely owner: steipete — The relevant workspace runtime, terminal, native VNC, and host-key bootstrap history is concentrated in commits authored and merged by this owner.
• Options:
  • Generate in guest and hand back public identity (recommended): Have first boot generate the host key, report only public identity/fingerprint over an authenticated readiness channel, and keep private key material out of provider metadata.
  • Use a non-retained secret channel: Keep coordinator-side generation only if maintainers can provide a provider-independent delivery path that is not persisted in launch metadata.
  • Document provider metadata as trusted: Accept the current exposure model only if maintainers decide provider user-data readers are inside the workspace trust boundary.

Security
Needs attention: The issue raises a concrete generated-credential exposure path in current coordinator workspace provisioning.

Review details

Best possible solution:

Keep workspace host-key pinning, but stop placing generated host private keys in provider-retained metadata and add regression coverage for cloud-init, AWS user-data, Azure customData, and GCP metadata.

Do we have a high-confidence way to reproduce the issue?

Yes for source reproduction: current main clearly generates the workspace host key, renders the private half into cloud-init, and sends that bootstrap through provider metadata paths. I did not perform a live provider launch.

Is this the best way to solve the issue?

Yes on direction: keeping host-key pinning while removing private key material from retained provider metadata is the right security boundary. The exact handoff mechanism needs maintainer security design before implementation.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The code path is source-reproducible, but I did not run a live AWS/Azure/GCP launch to inspect provider-side metadata after provisioning.
• A durable fix likely needs a maintainer-owned security design for guest-generated host identity or another non-retained secret-delivery channel.

Codex review notes: model internal, reasoning high; reviewed against 52d272fbc45e.

Label changes

Label changes:

• add P2: This is a concrete security-boundary bug with generated credential exposure, but the impact is bounded by provider metadata read access, workspace lifetime, and later network positioning.
• add impact:security: The issue concerns generated SSH host private key material and the integrity of workspace SSH host-key verification.
• add issue-rating: 🦞 diamond lobster: Current issue advisory state selects this label.
• add clawsweeper:source-repro: Current issue advisory state selects this label.
• add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.
• add clawsweeper:fix-shape-clear: Current issue advisory state selects this label.
• add clawsweeper:needs-maintainer-review: Current issue advisory state selects this label.
• add clawsweeper:needs-product-decision: Current issue advisory state selects this label.
• add clawsweeper:needs-security-review: Current issue advisory state selects this label.

Label justifications:

• P2: This is a concrete security-boundary bug with generated credential exposure, but the impact is bounded by provider metadata read access, workspace lifetime, and later network positioning.
• impact:security: The issue concerns generated SSH host private key material and the integrity of workspace SSH host-key verification.

Evidence reviewed

Security concerns:

• [medium] Generated SSH host private key reaches provider metadata — worker/src/bootstrap.ts:697
cloudInit renders sshHostPrivateKey as ssh_keys.ed25519_private, and AWS, Azure, and GCP send the same bootstrap through provider metadata/user-data channels.
  Confidence: 0.93

Acceptance criteria:

• npm test --prefix worker -- bootstrap
• npm test --prefix worker -- fleet
• npm test --prefix worker -- aws
• npm test --prefix worker -- azure
• npm test --prefix worker -- gcp

What I checked:

• Current source generates and forwards host private key: Workspace provisioning generates an Ed25519 host key pair, passes it into workspaceLeaseRequest, base64-encodes the private key into request headers, and later copies it into LeaseConfig.sshHostPrivateKey. (worker/src/fleet.ts:3353, 52d272fbc45e)
• Current bootstrap renders private key into cloud-init: cloudInit includes optionalSSHHostKeys, and that helper renders config.sshHostPrivateKey under ssh_keys.ed25519_private. (worker/src/bootstrap.ts:688, 52d272fbc45e)
• Provider adapters send that bootstrap through metadata paths: AWS sends awsRunInstancesUserData(config) as EC2 UserData, Azure base64-encodes cloudInit(config) into osProfile.customData, and GCP writes cloudInit(config) to metadata item user-data. (worker/src/aws.ts:1162, 52d272fbc45e)
• Host key verification is security-relevant: Workspace SSH connection setup pins the expected SHA-256 host-key fingerprint, so disclosure of the matching private key undermines the verifier for later workspace SSH-backed connections. (worker/src/fleet.ts:13671, 52d272fbc45e)
• Existing tests assert the current unsafe behavior: bootstrap.test.ts expects ed25519_private in cloud-init, and fleet.test.ts expects workspace lease creation to provide an OpenSSH private host key to provider config. (worker/test/bootstrap.test.ts:66, 52d272fbc45e)
• Security policy boundary matches the report: SECURITY.md lists generated credentials exposed beyond the current OS user by Crabbox as in scope, while docs/operations.md documents the coordinator-generated workspace host identity and that the workspace SSH private key should remain with the coordinator. (SECURITY.md:43, 52d272fbc45e)

Likely related people:

• steipete: Authored and merged the workspace runtime adapter, workspace terminal support, native VNC workspace grants, and the cloud-init host-key installation path that the report traces. (role: feature owner and recent area contributor; confidence: high; commits: 9e208c80cd1a, c86d33972cb5, 3a3b9798493d; files: worker/src/fleet.ts, worker/src/bootstrap.ts, worker/test/fleet.test.ts)
• vincentkoc: Recently merged coordinator-managed lease work in worker/src/fleet.ts; this is adjacent to the same coordinator/provider lease surface but not the specific host-key path. (role: recent adjacent contributor; confidence: low; commits: caa79b1eb5c8; files: worker/src/fleet.ts)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.