Spike: Local dev against real Foundry datasets (auth + workflow)

[anand-testcompare]

Overview

Enable a local developer workflow that runs the same pipeline mode codepaths as Foundry, but against real Foundry datasets/streams.

This is primarily a question of auth, safe defaults, and developer ergonomics so there are minimal surprises when moving from local to Foundry remote.

Problem Statement

We can already run Foundry shaped execution locally via the mock harness (container, BUILD2_TOKEN, RESOURCE_ALIAS_MAP, Foundry HTTP client). The missing piece is a supported way to:

• obtain a valid bearer token locally with the right permissions and lifecycle
• supply Foundry dataset aliases and branches safely
• run locally without accidentally writing to production datasets/streams
• understand what fidelity gaps remain vs a true Foundry pipeline build

Non Goals

• Build a full Foundry SDK
• Implement a complex interactive login UX unless it is clearly required

Auth Options To Evaluate

1. Manual user token workflow

   • Developer creates a token and writes it into a local file used as BUILD2_TOKEN.
   • Pros: simple, no app registration.
   • Cons: permissions differ from pipeline build token; token rotation; operational footguns.

2. OAuth2 client credentials for a service identity

   • Register an OAuth app, use client credentials to mint short lived tokens.
   • Pros: closest to a "service" model; can be least privilege.
   • Cons: setup overhead; secret handling.

3. OAuth2 authorization code for a developer identity

   • Local CLI flow that mints tokens, optionally with refresh.
   • Pros: good UX once built.
   • Cons: setup and maintenance; still differs from pipeline build token semantics.

Deliverables

• A recommended approach with tradeoffs and required Foundry configuration.
• A concrete "local run against real datasets" recipe.
• Guardrails for safety.

Exit Criteria

• We can run the container locally in foundry mode against a real stack by providing:
  • FOUNDRY_URL
  • a BUILD2_TOKEN file
  • a RESOURCE_ALIAS_MAP file
• We document the constraints and the remaining non parity areas (egress policy, build opened transactions, etc).

Implementation Sketch (If We Build Anything)

• ./dev foundry-alias-map command that generates a RESOURCE_ALIAS_MAP JSON from flags, to reduce mistakes.
• Optional: ./dev foundry-token helper that writes a token to a file (depending on chosen auth approach).
• A safety interlock requiring --i-understand-this-writes-to-foundry or a dedicated --allow-write flag unless output RID matches a configured allowlist.

Acceptance Criteria

• Documented workflow exists and is validated against a real stack.
  • Test: manual checklist
• Any code changes have unit tests for parsing and safety checks.
  • Test: unit

Open Questions

• What is the least painful token acquisition mechanism allowed in the target Foundry environments.
• Is external access to the stack hostname permitted from developer machines.
• Should local runs be read only by default.