Skip to content

workspace resolve 404: workspace create bypasses Cloud, orphaning the key it stores #1260

Description

@khaliqgant

Summary

agent-relay workspace create / agentworkforce login --workspace <key> mints a workspace directly against Relaycast and never creates the corresponding Cloud Postgres row, so agent-relay workspace active / agentworkforce deploy immediately 404s ("Workspace not found") on that same key. This is the recurring "dev vs prod workspace" resolve failure seen across cloud, agents, workforce, and internal-agents sessions since at least 2026-07-04.

Reproduced today in internal-agents:

agentworkforce deploy market-intel/reddit-monitor/persona.json --mode cloud --on-exists update --input SLACK_CHANNEL=C0ADJH54N3T
agentworkforce deploy failed: Workspace resolve failed at /api/v1/workspaces/active?key=rk_live_c590d2bc...: 404 Workspace not found

The failing key is the locally cached default workspace key in ~/.agentworkforce/relay/workspaces.json — it was never created through Cloud's unified endpoint, so it has no relay_workspaces row to resolve.

Root cause (confirmed via live reproduction 2026-07-13, full report below)

agent-relay workspace create creates a messaging-only workspace directly in Relaycast (RelayCast.createWorkspace, relay/packages/sdk/src/agent-relay.ts:127-150) and stores only the returned rk_live_ key locally. agent-relay workspace active (used by agentworkforce login/deploy) resolves that key through Cloud's resolveActiveWorkspace (relay/packages/cloud/src/workspaces.ts:342-379), whose primary lookup is getRelayWorkspaceByRelaycastApiKey against Cloud Postgres. The direct-Relaycast creation path never inserts that row, so Cloud correctly 404s even though the key is valid in Relaycast (GET https://gateway.relaycast.dev/v1/workspace returns 200 for the same key).

This mismatch was introduced by commit 0f87d23be, which added the Cloud-backed workspace active command alongside the pre-existing direct-Relaycast workspace create command without updating create or adding a create→active integration test.

Confirmed not the same bug as #2561/#2565: those fix row drift/partial-failure inside Cloud's unified create path. A key created via agent-relay workspace create never enters that path at all, so there's no row for #2565's reconciliation to act on. Live positive control: Cloud's own unified create→resolve returns 201→200 (proving #2565 is deployed and working), while a fresh agent-relay workspace create key remains Relaycast-valid and Cloud-404 afterward.

Fix recommendation

Primary fix belongs in AgentWorkforce/relay:

  1. Change relay/packages/cli/src/cli/commands/workspace.ts so workspace create calls @agent-relay/cloud.createWorkspace (the unified Cloud endpoint), not AgentRelay.createWorkspace/RelayCast.createWorkspace (direct Relaycast-only).
  2. Extend WorkspaceCreateResponse / normalizeWorkspaceCreateResponse (relay/packages/cloud/src/types.ts, workspaces.ts) to retain the Cloud response's relaycastApiKey, and store that via setWorkspaceKey(name, key).
  3. Align the CLI option name with Cloud (--api-url) rather than the current Relaycast-only --base-url; fail clearly when no Cloud session exists.
  4. Add a regression test at the CLI boundary: authenticated workspace create must call Cloud /api/v1/workspaces, save its returned Relaycast key, and an immediate workspace active --json must resolve it successfully. Assert it does NOT invoke direct RelayCast.createWorkspace.
  5. Keep AgentRelay.createWorkspace available as an explicit messaging-only SDK primitive if that's still wanted publicly — the bug is using it for a CLI command whose stored workspace is later advertised as canonical Cloud/Relayfile state.

Existing orphaned keys

Fixing the CLI prevents new orphans but doesn't repair keys already stored locally (e.g. this machine's default, mini-sandbox-team, and others in ~/.agentworkforce/relay/workspaces.json). Needs a separate, explicitly-authenticated migration/adoption command that validates the presented key via Relaycast, provisions a canonical Cloud/Relayfile workspace, and persists both the plaintext key and the Relaycast provider workspace ID under the authenticated Cloud user. Should not silently auto-create rows from an unauthenticated resolver request.

Repository ownership

Repository Role Verdict
relay Direct create + local key store + Cloud-backed active resolver Primary fix owner
cloud Canonical unified create/resolve; optional orphan-adoption endpoint Working correctly for its own create path
relaycast-cloud Messaging workspace/key creation and auth Healthy; not the missing-row owner
relayfile Calls workspace active, then requests delegated credentials Downstream victim; its own setup path already uses the correct Cloud create
relayfile-cloud Relayfile data plane, post-resolution Not involved

Full investigation report

A full reproduction (curl traces against all 3 resolver candidates, live positive/negative controls, exact file:line references) is available — happy to attach/paste on request. Investigated 2026-07-13 by an autonomous relay investigation session (019f5b95-8749-7da0-a54b-8c5f4678d8fe); reproduced again independently today (2026-07-14) hitting the exact same failure from internal-agents.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions