Consolidate grove access behind one substrate-agnostic exec/attach abstraction

[scuba10steve]

Summary

Grove access is currently implemented three separate times, with three different notions of "reach into a grove", all of them hardcoded to SSH-into-a-VM. Consolidate them behind a single substrate-agnostic exec/attach abstraction, and finish wiring the SSH gateway (#200) so user-facing access actually flows through it.

This is the prerequisite plumbing for the container-native providers in #86 — under all three of that issue's candidate architectures, per-substrate exec/attach has to exist before #87/#88 can be written.

Context: the three paths today

1. Control-plane exec (outbound, provisioning + management). SshExecutor / CommandRunner / SshCommandBuilder in nursery/, consumed by FruitGrower and DevcontainerCli.

2. Control-plane exec leaking into the API layer. SshExecutor is constructed directly inside the trellis service layer:

• trellis/src/main/java/dev/orchard/api/service/GroveService.java:259, :310, :343, :451
• trellis/src/main/java/dev/orchard/api/service/BeeService.java:229
• trellis/src/main/java/dev/orchard/trellis/config/QemuStartupRunner.java

Plus the default method that gives every provider SSH whether it wants it or not:

• nursery/src/main/java/dev/orchard/nursery/SeedlingProvider.java:29 — verifyDevcontainerCli(seedling, expectedVersion, new SshExecutor(seedling))

3. Gateway relay (inbound, user-facing). Added in #200 (96d1a50). GroveRelayServer + SeedlingRelay + RelayCommand authenticate an external user and pump shell/exec/sftp through one cached gateway→seedling SSH session.

Path 3 is the newest and is genuinely the right shape — GatewayRoute is already an indirection over "how do I reach this grove". But it resolves to an SSH endpoint on a VM and nothing else:

• SeedlingRelay.relaySession() dials route.seedlingIp() / route.seedlingPort() as DEFAULT_SSH_USER = "cultivator"
• GatewayGroveResponse is typed as seedlingIp + seedlingPort, documented as "Route info the SSH gateway needs to reach a grove's seedling"
• GatewayGroveService.isRoutable() requires seedling != null && ipAddress != null && state == SAPLING

A Docker/Podman or Kubernetes grove (#88, #87) has no sshd to relay to, so all three paths need a per-substrate strategy — docker exec, kubectl exec, ECS Exec/SSM — rather than one SSH implementation.

Considered and rejected: route ALL SSH through the gateway process at runtime

The obvious reading of "one choke-point" is to make trellis/nursery exec through the running gateway. Attractive — one audit point, one key store, identical semantics for platform and user traffic. It does not work, for three reasons, all load-bearing:

Bootstrap paradox. SeedlingProvider.verifyDevcontainerCli is documented as "called by each provider before transitioning a Seedling to SAPLING (READY)", and GroveService.java:158 calls it to decide whether the seedling becomes SAPLING. But GatewayGroveService.isRoutable() requires state == SAPLING. Provisioning-time exec is therefore strictly earlier than routability — routed through the gateway it could never resolve. cloudInitLogTail (GroveService.java:296) is a sharper case: it is diagnostic exec against a seedling that is by definition unhealthy, i.e. exactly when a readiness-gated route is unavailable.

Runtime cycle between deployables. The gateway already calls trellis to resolve routes (GroveResolver → TrellisApiClient → GatewayGroveController at /api/gateway). Adding trellis → gateway for exec makes the two processes mutually dependent at startup, and converts a gateway outage from an access outage into a provisioning outage. That is a strictly larger blast radius than the single-point-of-failure the consolidation was willing to accept.

Trust inversion. The gateway is an edge component whose purpose is authenticating untrusted external principals (OwnerTokenAuthenticator, KeyAuthenticator; AcceptAllServerKeyVerifier on the outbound leg). Trellis is already inside the trust boundary. Making the control plane authenticate to the edge to reach its own resources requires a credential structurally at least as privileged as any user's.

Recorded here so it does not get re-litigated. Consolidate the abstraction, not the runtime hop.

Proposed work

1. One exec/attach seam, shared by all three paths

Extract the "reach into a grove and run/attach something" strategy into a module both GatewayApplication and nursery/trellis depend on (:core, :nursery, or a new :access — see open questions). Requirements:

• Per-substrate implementations: SSH-to-VM (today), docker exec (Local Docker / Podman nursery provider #88), kubectl exec (Kubernetes nursery provider (generic) #87), later ECS Exec / nomad alloc exec
• Covers all three modes the gateway already needs: one-shot exec, interactive shell, and sftp/file transfer — the control plane only uses one-shot today, but a shared seam should not foreclose the others
• Works pre-readiness. The provisioning and diagnostic call sites run before SAPLING; the abstraction must be reachable without a readiness-gated route lookup
• Retires the new SshExecutor(seedling) default in SeedlingProvider.java:29 so a provider is not born SSH-shaped

2. Get SshExecutor out of the trellis service layer

GroveService and BeeService should depend on the seam, not construct SSH clients. This is mechanical once (1) exists and is worth doing regardless of #86's outcome.

3. Finish wiring the gateway

• Deployment. The gateway appears in no docker-compose.yml, .github/, or scripts/ reference, and no module declares project(":gateway"). It builds and is never run. Needs a compose service, a CI smoke test, and a release/deploy story alongside trellis.
• De-QEMU the trust root. SeedlingRelay.loadInternalKey() throws with a message naming QemuEnvironmentInitializer as the key's producer. Internal key provisioning must be provider-agnostic before any non-QEMU substrate can be relayed to.
• Generalize the route contract. GatewayGroveResponse(seedlingIp, seedlingPort) should carry a substrate handle, not an IP/port pair. Coordinate with the naming discussion in Expand nursery provider support to container-native substrates #86 ("Seedling" no longer fits when there is no VM).
• Point user-facing access at it. Interactive, post-provisioning, user-initiated access is the genuinely gateway-shaped subset — route trowel's user-facing access through the gateway instead of direct SSH, so platform and user traffic share one implementation without sharing one process.

Open questions

1. Module placement. :core (widest reach, but drags SSH/sshd deps into everything), :nursery (natural home for substrate knowledge, but the gateway would then depend on nursery), or a new :access module both sides depend on. Leaning toward the third.
2. Does the gateway keep its own relay implementation? The gateway's leg is a persistent bidirectional channel; the control plane's is one-shot request/response. Shared interface with two implementations, or one interface expressive enough for both?
3. Audit consolidation without runtime consolidation. A single audit trail for all grove access was a real motivation for the rejected design. If exec is a shared library rather than a shared service, where do audit events land — an event published to trellis, a shared appender, or per-process logs correlated by grove id?
4. Relationship to Nursery: support dynamic provider configuration #170. Dynamic provider configuration means the substrate backing a grove can change at runtime. Does the exec seam resolve a strategy per call, or cache per grove?
5. Ordering against Expand nursery provider support to container-native substrates #86. This is prerequisite plumbing, but Expand nursery provider support to container-native substrates #86's model-fit decision (virtualized Seedling / split providers / unified GroveProvider) determines what a "substrate handle" is. Do we land a minimal seam now and refine, or wait for the decision?

Out of Scope

• Choosing Expand nursery provider support to container-native substrates #86's provider architecture — tracked there.
• Implementing any specific container substrate (Kubernetes nursery provider (generic) #87, Local Docker / Podman nursery provider #88, AWS ECS / Fargate nursery provider #89, GCP Cloud Run nursery provider #90, Azure Container Apps / ACI nursery provider #91, HashiCorp Nomad nursery provider #92).
• Multi-IdP auth for the gateway's token path (fence: support multiple upstream IdPs (multi-IdP) #196).
• Session recording / replay as a product feature (audit plumbing is in scope; a recording feature is not).

Related

• Expand nursery provider support to container-native substrates #86 — container-native nursery providers (this unblocks it)
• Kubernetes nursery provider (generic) #87, Local Docker / Podman nursery provider #88 — first substrates needing non-SSH exec
• Nursery: support dynamic provider configuration #170 — dynamic provider configuration (open question 4)
• Reattach surviving instances via persisted (pid, startInstant) metadata + provider-agnostic reattach() hook #137 — provider-agnostic reattach() hook; same "providers rediscover their own resources" instinct
• fence: support multiple upstream IdPs (multi-IdP) #196 — fence multi-IdP, touches the gateway's FenceTokenClient auth path
• Client-registered SSH public keys + gateway for grove access (stop reusing trellis's internal key) #200 — the gateway PR this builds on