diff --git a/RELEASING.md b/RELEASING.md new file mode 100644 index 0000000..5fcd543 --- /dev/null +++ b/RELEASING.md @@ -0,0 +1,175 @@ +# Releasing Mantle + +Mantle uses [Changesets](https://github.com/changesets/changesets) for versioning and automated releases. +The CI pipeline is fully automated once the Version PR is merged — no manual tagging or publishing required. + +## Prerequisites + +Before performing a manual release (bootstrap or re-tag), confirm you have: + +- **Push access** to `main` and tags on `github.com/dvflw/mantle` +- **`gh` CLI** installed and authenticated (`gh auth status`) +- **GHCR write access** — your GitHub account must have write access to `ghcr.io/dvflw` (granted via org membership) + +The automated CI path (Version PR → merge) requires no local tooling beyond git. + +## How It Works + +``` +1. changeset file added during dev + ↓ +2. push to main → release-please.yml creates/updates "Version PR" + ↓ +3. merge Version PR + ↓ +4. release-tags.yml detects version bumps, pushes package-scoped tags + ↓ +5a. release-engine.yml → goreleaser: binaries + Docker + GitHub Release + Trivy +5b. release-helm.yml → helm push OCI + GitHub Release +``` + +> **Note on workflow naming:** `release-please.yml` is named after a tool we evaluated but didn't adopt — it actually runs the Changesets CLI. A rename to `changeset-version.yml` is tracked in [the relevant issue]. + +## Step 1: Add a Changeset (During Development) + +Every PR that changes user-visible behavior needs a changeset. Run this from the repo root: + +```bash +bun run changeset +``` + +The interactive CLI asks: +1. **Which packages changed?** Select from `@mantle/engine`, `@mantle/helm-chart`, `@mantle/site` +2. **Bump type per package:** `major` / `minor` / `patch` +3. **Summary:** One-line description of the change (appears in CHANGELOG) + +This creates a file like `.changeset/fuzzy-lions-dance.md`. Commit it alongside your code. + +> **`@mantle/site`:** changesets version the docs site but do not trigger a separate release workflow. The `site/v*` tag is pushed automatically but there is no equivalent `release-site.yml` — site deploys happen via the Astro/hosting pipeline. + +**When to use each bump type:** +- `patch` — bug fixes, internal refactors, docs updates +- `minor` — new features, new CLI commands, new config fields +- `major` — breaking changes (workflow YAML format, CLI flags, API shape) + +If a PR touches only CI, tests, or tooling with no user impact, skip the changeset. + +## Step 2: The Version PR + +After any push to `main` that contains pending changesets, the `release-please.yml` workflow automatically creates (or updates) a PR titled **"ci: version packages"**. + +This PR: +- Bumps `version` in each affected `package.json` (taking the highest bump across all pending changesets) +- Generates / appends to `packages/*/CHANGELOG.md` +- Deletes the consumed changeset files + +Review the PR to confirm the version bumps and changelog entries are correct. If more changesets land on `main` before you merge, CI updates the PR automatically. + +> **Loop guard:** the workflow has an `if` condition that skips runs where the triggering commit message starts with `"ci: version packages"`. This prevents the workflow from retriggering on its own Version PR merge commit. If you ever rename the commit message in the workflow, update the guard condition in `release-please.yml` to match. + +## Step 3: Merge the Version PR + +When you're ready to cut a release, **merge the Version PR**. That's the release trigger. + +## Step 4: Tags and Artifacts (Automated) + +After the Version PR merges, `release-tags.yml` fires and: + +1. Detects which `package.json` files changed version +2. If the engine version bumped, syncs `Chart.yaml appVersion` and commits it back to `main` +3. Pushes package-scoped tags: `engine/v`, `helm-chart/v`, `site/v` + +Those tags trigger the package-specific release workflows: + +| Tag pattern | Workflow | What it produces | +|---|---|---| +| `engine/v*` | `release-engine.yml` | goreleaser builds Linux/macOS amd64/arm64 binaries + checksums + LICENSE, pushes multi-arch `ghcr.io/dvflw/mantle:` Docker image (+ floating tags on stable), creates GitHub Release, runs Trivy CVE scan | +| `helm-chart/v*` | `release-helm.yml` | Packages and pushes `oci://ghcr.io/dvflw/helm-charts/mantle:`, creates GitHub Release | + +**How the engine tag translates to a clean version:** the workflow strips the `engine/` prefix, sets `GORELEASER_CURRENT_TAG=v`, and creates a local `git tag v` alias so goreleaser can validate the tag against the current commit. Only the namespaced tag (`engine/v`) is permanent in the remote. + +**Floating tags** (`major.minor`, `major`, `latest`) are only pushed for stable releases. For pre-release versions (e.g. `0.5.0-rc.1`) the versioned image is pushed but floating tags are skipped. + +**Platform-specific image tags:** the pipeline no longer publishes per-arch tags like `ghcr.io/dvflw/mantle:-amd64`. Only the multi-arch manifest tag (`ghcr.io/dvflw/mantle:`) is pushed. Update any CI or deploy configs that reference the old suffixed tags. + +**Trivy scan policy:** the `trivy` job runs after `release` with `exit-code: 1` on `CRITICAL` or `HIGH` CVEs. A failure marks the workflow run as failed but does not retract the already-published release. If a CVE scan fails post-release, open a patch release issue immediately. + +**Partial-failure recovery:** if `release-tags.yml` fails after pushing some but not all tags, push the missing tags manually: + +```bash +git tag helm-chart/v +git push origin helm-chart/v +``` + +The tag-triggered workflows are safe to retrigger — goreleaser will fail cleanly if the GitHub Release already exists. + +## First Release of a New Version (No Pending Changesets) + +This applies when `package.json` is already at the target version but `.changeset/` contains no pending changeset files — for example, the first-ever release of a newly bootstrapped repo, or after manually editing `package.json` outside the changeset flow. + +In this state the `release-please.yml` workflow has nothing to process and will not create a Version PR. Push the tags directly: + +```bash +git tag engine/v +git push origin engine/v + +# If also releasing the Helm chart at the same version: +git tag helm-chart/v +git push origin helm-chart/v +``` + +After the initial tag, use the full changeset flow for all subsequent releases. + +## Verifying a Release + +After the workflows complete, substitute your version for ``: + +```bash +# Check GitHub Releases +gh release list + +# Verify Docker image +docker pull ghcr.io/dvflw/mantle: +docker run --rm ghcr.io/dvflw/mantle: mantle version + +# Verify Helm chart +helm show chart oci://ghcr.io/dvflw/helm-charts/mantle --version +``` + +## Pre-releases + +To cut a pre-release (e.g. `0.5.0-rc.1`), no changeset is needed — bump the version manually and tag directly: + +```bash +# 1. Set the pre-release version in package.json +# Edit packages/engine/package.json: "version": "0.5.0-rc.1" +git add packages/engine/package.json +git commit -m "chore: bump engine to 0.5.0-rc.1" +git push origin main + +# 2. Tag and push +git tag engine/v0.5.0-rc.1 +git push origin engine/v0.5.0-rc.1 +``` + +goreleaser's `prerelease: auto` publishes the GitHub Release as a pre-release when the version contains a pre-release identifier. The versioned Docker image (`0.5.0-rc.1`) is pushed; floating tags (`latest`, `major`, `major.minor`) are not. + +## Rollback + +Releases are immutable — GitHub Releases and pushed OCI images cannot be unpublished retroactively for users who have already pulled them. Rolling back means directing users to the last known-good version and cutting a patch. + +**1. Communicate immediately** with the last known-good version and the broken version to avoid. + +**2. Optionally hide the broken release from the GitHub Releases UI:** + +```bash +# Mark as pre-release so it no longer shows as the "latest" release +gh release edit engine/v --prerelease + +# Or move to draft (still accessible by direct URL/tag, just hidden from listing) +gh release edit engine/v --draft +``` + +> These do not retract published Docker images or Helm charts from GHCR. Downstream users who have already pulled the image are not affected. + +**3. Cut a patch release** as the authoritative fix via the normal changeset flow. diff --git a/docs/superpowers/specs/2026-03-18-ci-pipeline-design.md b/docs/superpowers/specs/2026-03-18-ci-pipeline-design.md new file mode 100644 index 0000000..3b47860 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-ci-pipeline-design.md @@ -0,0 +1,72 @@ +# Design: CI/CD Pipeline — test, vet, lint + +> Linear issue: [DVFLW-275](https://linear.app/dvflw/issue/DVFLW-275/cicd-pipeline-test-vet-lint) +> Date: 2026-03-18 + +## Goal + +Add GitHub Actions CI pipeline that runs tests, vet, and lint on every push and PR. Add golangci-lint configuration. Add CI status badge to README. + +## Acceptance Criteria + +- `go test ./...` runs on every push and PR +- `go vet ./...` and `golangci-lint` run on every push and PR +- Integration tests run against Postgres via testcontainers (when added) +- CI fails on lint errors or test failures + +## Files + +- Create: `.github/workflows/ci.yml` +- Create: `.golangci.yml` +- Modify: `README.md` — add CI badge + +## GitHub Actions Workflow + +`.github/workflows/ci.yml` — triggered on push and pull_request to all branches. + +Three parallel jobs on `ubuntu-latest` with Go 1.24: + +### test +- Checkout, setup Go +- `go test -race ./...` +- Docker available by default for future testcontainers tests + +### vet +- Checkout, setup Go +- `go vet ./...` + +### lint +- Checkout, setup Go +- Uses `golangci/golangci-lint-action` with pinned version + +## golangci-lint Config + +`.golangci.yml` — conservative linter set that catches real bugs: + +```yaml +run: + timeout: 5m + +linters: + enable: + - errcheck + - govet + - staticcheck + - unused + - ineffassign + - gosimple +``` + +## README Badge + +Add CI status badge at the top of README.md, right after the `# Mantle` heading: + +```markdown +[![CI](https://github.com/dvflw/mantle/actions/workflows/ci.yml/badge.svg)](https://github.com/dvflw/mantle/actions/workflows/ci.yml) +``` + +## What's NOT Included + +- Release/deploy workflows (Phase 4) +- Code coverage reporting +- Go module caching (add later if CI is slow) diff --git a/docs/superpowers/specs/2026-03-18-config-system-design.md b/docs/superpowers/specs/2026-03-18-config-system-design.md new file mode 100644 index 0000000..56de5d2 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-config-system-design.md @@ -0,0 +1,147 @@ +# Design: Config System — mantle.yaml and CLI Flags + +> Linear issue: [DVFLW-272](https://linear.app/dvflw/issue/DVFLW-272/config-system-mantleyaml-and-cli-flags) +> Date: 2026-03-18 + +## Goal + +Add a configuration system using Viper that loads settings from `mantle.yaml`, with environment variable and CLI flag overrides. + +## Acceptance Criteria + +- `mantle.yaml` supports Postgres connection string, API listen address, log level +- CLI flags override config file values +- Env vars (e.g., `MANTLE_DATABASE_URL`) override both config file and defaults +- Config file path configurable via `--config` flag + +## Package Structure + +```text +internal/ + config/ + config.go # Config struct, Load() function, Viper setup + config_test.go # Tests for loading, env overrides, flag overrides, defaults +``` + +## Config Struct + +```go +type Config struct { + Database DatabaseConfig `mapstructure:"database"` + API APIConfig `mapstructure:"api"` + Log LogConfig `mapstructure:"log"` +} + +type DatabaseConfig struct { + URL string `mapstructure:"url"` +} + +type APIConfig struct { + Address string `mapstructure:"address"` +} + +type LogConfig struct { + Level string `mapstructure:"level"` +} +``` + +## Default Values + +| Key | Default | Env Var | +|-----|---------|---------| +| `database.url` | `postgres://localhost:5432/mantle?sslmode=disable` | `MANTLE_DATABASE_URL` | +| `api.address` | `:8080` | `MANTLE_API_ADDRESS` | +| `log.level` | `info` | `MANTLE_LOG_LEVEL` | + +## Example mantle.yaml + +```yaml +database: + url: "postgres://localhost:5432/mantle?sslmode=disable" + +api: + address: ":8080" + +log: + level: "info" +``` + +## Load Function + +`Load(configPath string) (*Config, error)` — takes the `--config` flag value. + +Steps: +1. Set defaults for all keys +2. If `configPath` is provided, read that file; otherwise search current directory for `mantle.yaml` +3. Set `MANTLE_` env prefix via `SetEnvPrefix("MANTLE")` +4. Explicitly bind env vars with `BindEnv` for each key (not `AutomaticEnv`, which cannot reliably resolve nested keys with underscore delimiters): + - `viper.BindEnv("database.url", "MANTLE_DATABASE_URL")` + - `viper.BindEnv("api.address", "MANTLE_API_ADDRESS")` + - `viper.BindEnv("log.level", "MANTLE_LOG_LEVEL")` +5. Unmarshal into `Config` struct and return + +### Missing Config File Behavior + +- If `--config` is provided and the file does not exist: **hard error** (user explicitly asked for this file) +- If `--config` is not provided and no `mantle.yaml` in current directory: **silent fallback to defaults** (config file is optional) + +### Validation + +`Load()` returns an error only for YAML parse failures and type mismatches. It does **not** perform semantic validation (e.g., whether `database.url` is a valid Postgres connection string). Semantic validation belongs in the consuming code. + +## Override Precedence + +Viper's built-in precedence (highest to lowest): +1. CLI flags (bound to Viper keys via `BindPFlag`) +2. Environment variables (`MANTLE_` prefix, explicit `BindEnv` calls) +3. Config file (`mantle.yaml`) +4. Defaults + +## CLI Flags + +Add these persistent flags on the root command: +- `--database-url` — binds to Viper key `database.url` +- `--api-address` — binds to Viper key `api.address` +- `--log-level` — binds to Viper key `log.level` + +Flags are bound to Viper keys via `viper.BindPFlag()` inside `Load()`, which receives the `*cobra.Command` to access its flags. Updated signature: + +`Load(cmd *cobra.Command) (*Config, error)` — reads `--config` flag from `cmd`, binds other flags to Viper. + +## Cobra Integration + +In `internal/cli/root.go`: +- Add `--database-url`, `--api-address`, `--log-level` as persistent flags +- `PersistentPreRunE` on the root command calls `config.Load(cmd)` which reads `--config`, binds flags, and loads config +- The resulting `*Config` is stored on the command's context via `context.WithValue` + +### Context Key + +Defined in `internal/config/`: + +```go +type contextKey struct{} + +// WithContext returns a new context with the config attached. +func WithContext(ctx context.Context, cfg *Config) context.Context { + return context.WithValue(ctx, contextKey{}, cfg) +} + +// FromContext retrieves the config from context. Returns nil if not set. +func FromContext(ctx context.Context) *Config { + cfg, _ := ctx.Value(contextKey{}).(*Config) + return cfg +} +``` + +The context key type is unexported, preventing collisions. Helper functions are in the `config` package so subcommands depend on `config` (not the other way around). + +## Dependencies + +- `github.com/spf13/viper` — config loading, env var binding, YAML parsing (must be added to go.mod) + +## What's NOT Included + +- Semantic validation of config values (belongs in consuming code) +- Hot reload / file watching +- Config fields beyond database.url, api.address, log.level (added by future phases) diff --git a/docs/superpowers/specs/2026-03-18-docker-compose-design.md b/docs/superpowers/specs/2026-03-18-docker-compose-design.md new file mode 100644 index 0000000..177bfc3 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-docker-compose-design.md @@ -0,0 +1,80 @@ +# Design: Docker-Compose for Local Development + +> Linear issue: [DVFLW-273](https://linear.app/dvflw/issue/DVFLW-273/docker-composeyml-for-local-development) +> Date: 2026-03-18 + +## Goal + +Set up docker-compose with Postgres for local development, update Makefile with dev workflow targets, and update README with setup instructions. + +## Acceptance Criteria + +- `docker-compose up` starts Postgres with correct credentials +- Makefile has common operations: migrate, test, build, lint, run +- README documents local dev setup + +## Files + +- Create: `docker-compose.yml` +- Modify: `Makefile` — add `migrate`, `run`, `dev` targets +- Modify: `README.md` — update Development section +- Modify: `internal/config/config.go` — update default database URL to match docker-compose credentials + +## docker-compose.yml + +```yaml +services: + postgres: + image: postgres:16-alpine + ports: + - "5432:5432" + environment: + POSTGRES_USER: mantle + POSTGRES_PASSWORD: mantle + POSTGRES_DB: mantle + volumes: + - pgdata:/var/lib/postgresql/data + +volumes: + pgdata: +``` + +Postgres 16 Alpine for small image size. Credentials: `mantle/mantle` on database `mantle`. Named volume for data persistence across restarts. + +## Config Default Update + +Change the default `database.url` from: +```text +postgres://localhost:5432/mantle?sslmode=disable +``` +to: +```text +postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable +``` + +This matches the docker-compose credentials so `make build && ./mantle` works out of the box against `docker-compose up`. + +Update the corresponding test expectations in `internal/config/config_test.go`. + +## Makefile Additions + +Add to existing Makefile (which already has `build`, `test`, `lint`, `clean`): + +- `migrate` — prints "No migrations yet. Run 'mantle init' when available." (placeholder for future use) +- `run` — `go run ./cmd/mantle $(ARGS)` for quick dev iteration without building +- `dev` — `docker-compose up -d` shorthand + +## README Updates + +Update the Development section to include: +1. Prerequisites (Go, Docker) +2. `docker-compose up -d` to start Postgres +3. `make build` / `make test` / `make lint` + +Keep it brief — the README already has most of this, just needs minor cleanup. + +## What's NOT Included + +- Database migrations (no schema yet — comes with workflow engine issues) +- Health check wait scripts for Postgres readiness +- Additional services (Redis, message queues, etc.) diff --git a/docs/superpowers/specs/2026-03-18-go-project-scaffold-design.md b/docs/superpowers/specs/2026-03-18-go-project-scaffold-design.md new file mode 100644 index 0000000..154cbec --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-go-project-scaffold-design.md @@ -0,0 +1,87 @@ +# Design: Go Project Scaffold and CLI Framework + +> Linear issue: [DVFLW-218](https://linear.app/dvflw/issue/DVFLW-218/go-project-scaffold-and-cli-framework) +> Date: 2026-03-18 + +## Goal + +Initialize the Go project with module, directory structure, CLI framework (Cobra), and a working `mantle version` command. This is the foundation everything else builds on. + +## Acceptance Criteria + +- `mantle version` prints version info (version, commit, build date) +- Project structure follows Go conventions (`cmd/`, `internal/`) +- CLI framework supports subcommands with help text + +## Project Structure + +```text +cmd/mantle/ + main.go # Entrypoint: initializes root command, calls Execute() +internal/ + cli/ + root.go # Root cobra command, global flags (--config) + version.go # `mantle version` subcommand + version/ + version.go # Version vars (Version, Commit, Date) set via ldflags +``` + +Only what's needed for `mantle version`. Future issues add packages under `internal/` as needed (engine/, config/, workflow/, connector/, secret/, api/, audit/). + +## Module Path + +`github.com/dvflw/mantle` + +## Version Output + +Default (dev build): +```text +mantle dev (none, built 2026-03-18T00:00:00Z) +``` + +With ldflags (release build): +```text +mantle v0.1.0 (abc1234, built 2026-03-18T15:30:00Z) +``` + +Three variables injected via `-ldflags -X`: +- `Version` — git tag or `dev` +- `Commit` — git short SHA or `none` +- `Date` — RFC3339 build timestamp + +## Build + +Plain dev build: +```bash +go build ./cmd/mantle +``` + +Release build (Makefile handles this): +```bash +go build -ldflags "-X github.com/dvflw/mantle/internal/version.Version=$(git describe --tags) \ + -X github.com/dvflw/mantle/internal/version.Commit=$(git rev-parse --short HEAD) \ + -X github.com/dvflw/mantle/internal/version.Date=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + ./cmd/mantle +``` + +## Makefile + +Targets: +- `build` — builds binary with git info injected via ldflags +- `test` — runs `go test ./...` +- `lint` — runs `golangci-lint run` +- `clean` — removes built binary + +## Dependencies + +- `github.com/spf13/cobra` — CLI framework (per CLAUDE.md) + +No other dependencies. + +## What's NOT Included + +- Config loading (`mantle.yaml`) — DVFLW-272 +- docker-compose — DVFLW-273 +- Health endpoints — DVFLW-274 +- CI pipeline — DVFLW-275 +- Any subcommands beyond `version` — added by their respective issues diff --git a/docs/superpowers/specs/2026-03-18-health-endpoints-design.md b/docs/superpowers/specs/2026-03-18-health-endpoints-design.md new file mode 100644 index 0000000..fb3f789 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-health-endpoints-design.md @@ -0,0 +1,106 @@ +# Design: Health Endpoints — /healthz and /readyz + +> Linear issue: [DVFLW-274](https://linear.app/dvflw/issue/DVFLW-274/health-endpoints-healthz-and-readyz) +> Date: 2026-03-18 + +## Goal + +Add `/healthz` and `/readyz` HTTP handlers for Kubernetes liveness and readiness probes. Also establish the `database/sql` connection layer used by readiness checks and all future database operations. + +## Acceptance Criteria + +- `/healthz` returns 200 when process is alive +- `/readyz` returns 200 when Postgres connection is healthy +- No authentication required on health endpoints + +## Package Structure + +``` +internal/ + db/ + db.go # Open() returns *sql.DB, context helpers + db_test.go # Unit test (skip if no Postgres) + api/ + health/ + health.go # /healthz and /readyz handlers + health_test.go # Tests using httptest +``` + +## internal/db/db.go + +```go +// Open connects to Postgres using the given URL and verifies the connection. +func Open(databaseURL string) (*sql.DB, error) +``` + +- Uses `pgx` driver (`github.com/jackc/pgx/v5/stdlib`) — `lib/pq` is in maintenance mode; `pgx` is the actively maintained Postgres driver for Go +- Calls `db.Ping()` to verify connection on open +- Returns `*sql.DB` with default pool settings (no tuning for now) + +Context helpers (same pattern as config): + +```go +type contextKey struct{} + +func WithContext(ctx context.Context, database *sql.DB) context.Context + +// FromContext retrieves the *sql.DB from context. Returns nil if not set. +func FromContext(ctx context.Context) *sql.DB +``` + +## internal/api/health/health.go + +Two handler constructors: + +### HealthzHandler + +```go +func HealthzHandler() http.HandlerFunc +``` + +Always returns HTTP 200 with `{"status":"ok"}`. Content-Type: `application/json`. + +### ReadyzHandler + +```go +func ReadyzHandler(database *sql.DB) http.HandlerFunc +``` + +If `database` is nil, returns 503 immediately without calling Ping. Otherwise calls `database.PingContext(r.Context())`: +- Success: HTTP 200 with `{"status":"ok"}` +- Failure: HTTP 503 with `{"status":"unavailable"}` + +Content-Type: `application/json`. + +## DB Connection Strategy + +**DB connection is NOT in `PersistentPreRunE`.** Many commands (`validate`, `plan`, `version`) are offline and should not require Postgres. Instead: + +- `internal/db/` provides `Open()` as a standalone function +- Commands that need a database call `db.Open()` in their own `RunE` and store on context if needed +- Future `mantle serve` will open the connection at startup and pass `*sql.DB` to health handlers and the engine + +For this issue, the health handlers receive `*sql.DB` as an explicit parameter. No Cobra integration changes needed — the handlers are tested standalone with `httptest` and wired into a server when `mantle serve` is built. + +### DB Lifecycle + +`*sql.DB` is closed by whichever code opened it. For CLI commands, close with `defer db.Close()` after `Open()`. For `mantle serve`, close on graceful shutdown. This is not a concern for this issue since we're only building handlers, not a running server. + +## Testing + +- `health_test.go` — uses `httptest.NewRecorder`: + - `/healthz` always returns 200 `{"status":"ok"}` + - `/readyz` with nil DB returns 503 + - `/readyz` with working DB returns 200 (tested via a real or in-memory approach) +- `db_test.go` — unit test for `Open()` with invalid URL returns error. Integration test against real Postgres skipped unless `MANTLE_TEST_DATABASE_URL` is set (real Postgres testing comes in DVFLW-278). + +## Dependencies + +- `github.com/jackc/pgx/v5` — Postgres driver for `database/sql` via `pgx/v5/stdlib` (must be added to go.mod) + +## What's NOT Included + +- CLI command to run health server (comes with `mantle serve` in Phase 5) +- Connection pooling configuration +- Integration tests against real Postgres (DVFLW-278) +- Authentication on health endpoints (explicitly excluded per acceptance criteria) diff --git a/docs/superpowers/specs/2026-03-18-migrations-design.md b/docs/superpowers/specs/2026-03-18-migrations-design.md new file mode 100644 index 0000000..09698da --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-migrations-design.md @@ -0,0 +1,193 @@ +# Design: Postgres Connection and Migration System + +> Linear issue: [DVFLW-219](https://linear.app/dvflw/issue/DVFLW-219/postgres-connection-and-migration-system) +> Date: 2026-03-18 + +## Goal + +Add a migration system using goose, create the initial Phase 1 schema (workflow_definitions, workflow_executions, step_executions), and add `mantle init`, `mantle migrate`, `mantle migrate status`, and `mantle migrate down` CLI commands. + +## Acceptance Criteria + +- `mantle init` runs migrations and creates the schema +- Migrations are versioned and idempotent +- Connection pooling configured appropriately + +## Package Structure + +``` +internal/ + db/ + db.go # Already exists — Open(), context helpers + migrate.go # Goose wrapper with embedded migrations + migrate_test.go # Tests with testcontainers + migrations/ + 001_initial_schema.sql # Creates all Phase 1 tables + cli/ + init.go # mantle init command + migrate.go # mantle migrate, migrate status, migrate down +``` + +## Migration Runner + +Uses `github.com/pressly/goose/v3` with SQL files embedded via `//go:embed`. + +### `internal/db/migrate.go` + +```go +// Migrate runs all pending migrations. +func Migrate(ctx context.Context, database *sql.DB) error + +// MigrateDown rolls back the last applied migration. +func MigrateDown(ctx context.Context, database *sql.DB) error + +// MigrateStatus returns migration status as formatted text. +func MigrateStatus(ctx context.Context, database *sql.DB) (string, error) +``` + +Implementation uses the **goose v3 provider API** (not the deprecated global-state functions): + +```go +//go:embed ../migrations/*.sql +var migrations embed.FS + +func newProvider(database *sql.DB) (*goose.Provider, error) { + return goose.NewProvider(goose.DialectPostgres, database, + migrations, goose.WithGoMigrations() /* SQL only */) +} +``` + +- `Migrate` creates a provider and calls `provider.Up(ctx)` +- `MigrateDown` creates a provider and calls `provider.Down(ctx)` +- `MigrateStatus` creates a provider, calls `provider.Status(ctx)` which returns `[]goose.MigrationStatus`, then formats the results into a human-readable string + +The provider API is instance-based (no global mutex), safe for concurrent test runs, and is the recommended approach since goose v3.16+. + +Note: `db.Open()` returns a stdlib-compatible `*sql.DB` via `pgx/v5/stdlib`, which is directly usable with goose. + +## Initial Migration: `001_initial_schema.sql` + +```sql +-- +goose Up +CREATE TABLE workflow_definitions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name TEXT NOT NULL, + version INTEGER NOT NULL, + content JSONB NOT NULL, + content_hash TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(name, version) +); + +CREATE TABLE workflow_executions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + workflow_name TEXT NOT NULL, + workflow_version INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'pending', + inputs JSONB, + started_at TIMESTAMPTZ, + completed_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE step_executions ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + execution_id UUID NOT NULL REFERENCES workflow_executions(id), + step_name TEXT NOT NULL, + attempt INTEGER NOT NULL DEFAULT 1, + status TEXT NOT NULL DEFAULT 'pending', + output JSONB, + error TEXT, + started_at TIMESTAMPTZ, + completed_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE(execution_id, step_name, attempt) +); + +-- +goose Down +DROP TABLE IF EXISTS step_executions; +DROP TABLE IF EXISTS workflow_executions; +DROP TABLE IF EXISTS workflow_definitions; +``` + +## CLI Commands + +All migration commands open their own DB connection in `RunE` and close with `defer`. Config is loaded by the root command's `PersistentPreRunE`. + +### `mantle init` + +Full onboarding command. Runs all pending migrations. Future-proofed to also handle any other first-time setup tasks. + +``` +$ mantle init +Running migrations... +OK 001_initial_schema.sql +Migrations complete. +``` + +### `mantle migrate` + +`migrate` is a Cobra parent command that is also runnable (has `RunE`). Running it bare executes migrations. It also has subcommands `status` and `down`. + +Runs all pending migrations. Semantically "upgrade" — same as init's migration step, but for ongoing upgrades. + +``` +$ mantle migrate +OK 002_add_indexes.sql +Migrations complete. +``` + +### `mantle migrate status` + +Shows applied and pending migrations. + +``` +$ mantle migrate status + Applied At Migration + ======================================= + 2026-03-18 20:00:00 +0000 001_initial_schema.sql + Pending 002_add_indexes.sql +``` + +### `mantle migrate down` + +Rolls back the last applied migration (single step). + +``` +$ mantle migrate down +Rolled back 001_initial_schema.sql +``` + +## DB Connection + +All commands use the existing `db.Open()` from `internal/db/db.go`. The `database.url` config value (from mantle.yaml, env var, or CLI flag) provides the connection string. + +No connection pooling changes for now — `database/sql` defaults are adequate. Pool tuning can be added to config when needed. + +## Testing + +- `migrate_test.go` — uses testcontainers to spin up real Postgres: + - Run `Migrate()`, verify all three tables exist via `information_schema.tables` + - Run `Migrate()` again, verify idempotent (no error, no duplicate tables) + - Run `MigrateDown()`, verify tables are dropped + - Skip if Docker is unavailable +- CLI tests — unit test that init/migrate commands call the right functions (using the existing test pattern of creating a root command and executing with args) + +## Dependencies + +- `github.com/pressly/goose/v3` — migration runner (must be added to go.mod) +- `github.com/testcontainers/testcontainers-go` — for integration tests (test dependency) +- `github.com/testcontainers/testcontainers-go/modules/postgres` — Postgres testcontainer module + +## Migration File Convention + +Use sequential numbering (`001_`, `002_`, etc.) for migrations. This is simpler than timestamp-based naming for a product with a single development stream. Future migrations follow the same pattern. + +## What's NOT Included + +- Migration generator (users can use `goose create` CLI directly) +- Seed data +- Indexes beyond primary keys and unique constraints (add when query patterns emerge) +- Connection pool tuning diff --git a/docs/superpowers/specs/2026-03-18-workflow-apply-design.md b/docs/superpowers/specs/2026-03-18-workflow-apply-design.md new file mode 100644 index 0000000..777d9a1 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-workflow-apply-design.md @@ -0,0 +1,106 @@ +# Design: Workflow Apply with Versioning — `mantle apply` + +> Linear issue: [DVFLW-229](https://linear.app/dvflw/issue/DVFLW-229/workflow-apply-with-versioning-mantle-apply) +> Date: 2026-03-18 + +## Goal + +Add `mantle apply` command that stores validated workflow definitions as immutable versioned records in Postgres, with change detection via content hashing. + +## Acceptance Criteria + +- `mantle apply workflow.yaml` stores a new versioned definition +- Each apply creates a new version; definitions are immutable once stored +- Applying an unchanged workflow reports "no changes" and does not create a new version +- Single-tenant in Phase 1 (no team scoping — comes in Phase 6) + +## Package Structure + +``` +internal/ + workflow/ + store.go # Database operations — Save, GetLatestVersion, GetLatestHash + store_test.go # Integration tests with testcontainers + cli/ + apply.go # mantle apply command +``` + +## Database Operations — `internal/workflow/store.go` + +Direct SQL against Postgres. No ORM, no Store interface. + +```go +// Save validates, hashes, and stores a workflow definition. +// Returns the new version number. Returns 0 if content is unchanged. +func Save(ctx context.Context, db *sql.DB, result *ParseResult, rawContent []byte) (int, error) + +// GetLatestVersion returns the latest version number for a workflow name. +// Returns 0 if no versions exist. +func GetLatestVersion(ctx context.Context, db *sql.DB, name string) (int, error) + +// GetLatestHash returns the content_hash of the latest version for a workflow name. +// Returns empty string if no versions exist. +func GetLatestHash(ctx context.Context, db *sql.DB, name string) (string, error) +``` + +### Save Flow + +1. Validate the workflow via `Validate(result)` — return error if validation fails +2. Compute SHA-256 hash of raw YAML bytes +3. Query latest content_hash for this workflow name +4. If hash matches latest → return 0 (no changes) +5. Query latest version number, increment by 1 (first version is 1) +6. Insert new row into `workflow_definitions`: + - `name`: workflow name + - `version`: incremented version number + - `content`: parsed Workflow struct serialized as JSONB + - `content_hash`: SHA-256 hex string of raw YAML +7. Return new version number + +### Content Storage + +The `content` JSONB column stores the parsed workflow struct as JSON (not raw YAML). This makes the content queryable via Postgres JSON operators and avoids re-parsing YAML at read time. The raw YAML is hashed for change detection but not stored separately. + +## CLI Command — `internal/cli/apply.go` + +`mantle apply ` — requires DB connection. + +- Config loaded by root's `PersistentPreRunE` +- Opens DB connection in `RunE`, closes with `defer` +- Reads raw file bytes for hashing +- Calls `workflow.Parse()` for parsing +- Calls `workflow.Save()` for validation + storage +- Reports result + +Output on new version: +``` +Applied fetch-and-summarize version 1 +``` + +Output on no changes: +``` +No changes — fetch-and-summarize is already at version 1 +``` + +## Testing + +`store_test.go` — testcontainers integration tests (reuses `setupTestDB` pattern from `internal/db/migrate_test.go`): + +- Save a workflow → verify returns version 1 +- Query `workflow_definitions` table → verify row exists with correct name, version, hash +- Save identical content again → verify returns 0 (no changes) +- Modify content, save again → verify returns version 2 +- Verify content_hash is correct SHA-256 of raw bytes + +Tests require running migrations first via `db.Migrate()`. + +## Dependencies + +No new dependencies. Uses `crypto/sha256` and `encoding/hex` from stdlib. + +## What's NOT Included + +- Team scoping (Phase 6) +- `mantle plan` diff output (DVFLW-228) +- Workflow deletion or listing +- Content validation beyond structural checks (CEL, connector params) diff --git a/docs/superpowers/specs/2026-03-18-workflow-validation-design.md b/docs/superpowers/specs/2026-03-18-workflow-validation-design.md new file mode 100644 index 0000000..4fb7754 --- /dev/null +++ b/docs/superpowers/specs/2026-03-18-workflow-validation-design.md @@ -0,0 +1,144 @@ +# Design: Offline Workflow Validation — `mantle validate` + +> Linear issue: [DVFLW-227](https://linear.app/dvflw/issue/DVFLW-227/offline-workflow-validation-mantle-validate) +> Date: 2026-03-18 + +## Goal + +Add workflow YAML parsing with line-number error reporting and structural validation. Add `mantle validate` CLI command for offline schema conformance checking. + +## Acceptance Criteria + +- `mantle validate workflow.yaml` checks schema conformance offline +- Reports errors with line numbers and descriptive messages +- Exits with non-zero code on validation failure +- No network calls or engine connection required + +## Package Structure + +``` +internal/ + workflow/ + workflow.go # Workflow, Step, Input, RetryPolicy structs + parse.go # Parse(filename) — YAML parsing with yaml.Node line tracking + validate.go # Validate(*ParseResult) — structural validation rules + parse_test.go # Parsing tests + validate_test.go # Validation tests + cli/ + validate.go # mantle validate command +``` + +## Workflow Structs — `internal/workflow/workflow.go` + +```go +type Workflow struct { + Name string `yaml:"name"` + Description string `yaml:"description"` + Inputs map[string]Input `yaml:"inputs"` + Steps []Step `yaml:"steps"` +} + +type Input struct { + Type string `yaml:"type"` + Description string `yaml:"description"` +} + +type Step struct { + Name string `yaml:"name"` + Action string `yaml:"action"` + Params map[string]any `yaml:"params"` + If string `yaml:"if"` + Retry *RetryPolicy `yaml:"retry"` + Timeout string `yaml:"timeout"` +} + +type RetryPolicy struct { + MaxAttempts int `yaml:"max_attempts"` + Backoff string `yaml:"backoff"` +} +``` + +## ValidationError + +```go +type ValidationError struct { + Line int // 0 if unknown + Column int // 0 if unknown + Field string // e.g., "steps[0].name" + Message string +} + +func (e ValidationError) Error() string { + // Format: "line:col: error: message (field)" +} +``` + +## Parser — `internal/workflow/parse.go` + +```go +type ParseResult struct { + Workflow *Workflow + Root *yaml.Node // preserved for line number lookups +} + +func Parse(filename string) (*ParseResult, error) +``` + +- Reads file contents +- Unmarshals using `gopkg.in/yaml.v3` with a two-pass approach: + 1. `yaml.Unmarshal` into `yaml.Node` tree to capture line/column positions + 2. Decode the node tree into the `Workflow` struct +- Returns parse errors with line numbers if YAML is malformed +- Preserves the `yaml.Node` tree in `ParseResult` for the validator to reference line numbers + +## Validator — `internal/workflow/validate.go` + +```go +func Validate(result *ParseResult) []ValidationError +``` + +Structural validation rules: +- `name` is required, non-empty, and matches `^[a-z][a-z0-9-]*$` (lowercase alphanumeric with hyphens, used as CLI args and DB keys) +- `steps` is required and has at least one entry +- Each step has a non-empty `name` matching `^[a-z][a-z0-9-]*$` (same format as workflow name) +- Each step has a non-empty `action` +- Step names are unique within the workflow +- Input names must be valid identifiers matching `^[a-z][a-z0-9_]*$` (used in CEL as `inputs.`) +- Input `type` is one of: `string`, `number`, `boolean` (if inputs are declared) +- Retry `backoff` is one of: `fixed`, `exponential` (if retry is set) +- Retry `max_attempts` is > 0 (if retry is set) +- `timeout` parses as a valid positive Go duration (if set) + +The validator walks the `yaml.Node` tree to find line/column positions for each error. + +## CLI Command — `internal/cli/validate.go` + +`mantle validate ` — offline, no DB, no network. + +- Has its own no-op `PersistentPreRunE` (like the version command) since it needs no config or DB +- Reads file path from args +- Calls `workflow.Parse()` then `workflow.Validate()` +- Prints errors to stderr in the format: `filename:line:col: error: message (field)` +- Exits 0 on success, 1 on validation failure + +Output format on success: +``` +workflow.yaml: valid +``` + +Output format on failure: +``` +workflow.yaml:3:1: error: step name is required (steps[0].name) +workflow.yaml:7:5: error: unknown input type "integer", must be string, number, or boolean (inputs.url.type) +``` + +## Dependencies + +- `gopkg.in/yaml.v3` — YAML parsing with line number support (already a transitive dep via viper, but add as direct) + +## What's NOT Included + +- CEL expression syntax validation (needs expression engine) +- Action/connector-specific parameter validation (needs connector registry) +- JSON Schema document generation (can add later for editor support) +- Trigger declaration validation (Phase 5) diff --git a/docs/superpowers/specs/2026-03-19-ai-tool-use-parallel-execution-design.md b/docs/superpowers/specs/2026-03-19-ai-tool-use-parallel-execution-design.md new file mode 100644 index 0000000..539d8bc --- /dev/null +++ b/docs/superpowers/specs/2026-03-19-ai-tool-use-parallel-execution-design.md @@ -0,0 +1,302 @@ +# Design: AI Tool Use & Parallel Execution (Phase 8) + +> Phase 8 of Mantle V1.1. Enables LLM steps to call tools during execution, and independent workflow steps to run in parallel. +> +> Builds on Phase 7's multi-node distribution primitives: orchestrator/worker model, SKIP LOCKED claiming, lease/reaper lifecycle, and sub-step rows via `parent_step_id`. See [Multi-Node Distribution Design](2026-03-19-multi-node-distribution-design.md) for those foundations. + +## 1. Design Decisions + +- **Implicit parallelism via `depends_on`** — steps declare dependencies, the orchestrator resolves the DAG and runs independent steps concurrently. No explicit parallel block syntax. +- **Sub-steps as first-class `step_executions` rows** — tool call results stored as child rows with `parent_step_id`. Free checkpointing, free multi-node distribution, and visible in `mantle logs`. +- **Full LLM response + tool result caching for crash recovery** — deterministic replay on crash. Worst case: one redundant LLM call. +- **Single-level tool use** — tools execute connectors, they cannot themselves declare tools. Data model supports future multi-level (child rows can have children) without rearchitecting. + +## 2. Parallel Execution via DAG Resolution + +### Workflow YAML syntax + +```yaml +steps: + - name: fetch_users + action: http/request + params: + url: "https://api.example.com/users" + + - name: fetch_orders + action: http/request + params: + url: "https://api.example.com/orders" + + - name: summarize + action: ai/completion + depends_on: [fetch_users, fetch_orders] + params: + prompt: "Summarize: {{ steps.fetch_users.output.body }} and {{ steps.fetch_orders.output.body }}" +``` + +`fetch_users` and `fetch_orders` have no dependencies — the orchestrator creates both as `pending` simultaneously, and workers execute them in parallel. `summarize` waits until both are `completed`. + +### Orchestrator DAG logic + +The orchestrator from Phase 7 changes minimally. Instead of creating steps sequentially, it: + +1. Builds an in-memory DAG from `depends_on` declarations. +2. On each iteration, finds steps where all dependencies are in `completed` or `skipped` status. +3. Creates `pending` rows for all ready steps at once. +4. Waits for any status change, then re-evaluates. + +### Implicit dependency detection + +If a step's CEL expressions reference `steps.foo.output`, `foo` is added as an implicit dependency even without an explicit `depends_on`. This is validated at `mantle validate` time. + +**Scanning scope**: The validator extracts step references from all CEL-evaluated fields: `params` values (including nested maps), `if:` conditions, and tool `params` inside AI steps. It walks YAML string values looking for `{{ ... }}` template expressions and `steps.` references within them. + +**Detection method**: Regex extraction of `steps\.(\w+)` patterns from CEL expression strings. This is deliberately simple — it does not use the CEL AST. Only static, literal step name references are detected. Dynamic references (e.g., step names constructed via string concatenation) are not detected and require explicit `depends_on`. + +**Merge behavior**: Implicit dependencies are unioned with explicit `depends_on`. If a step both declares `depends_on: [foo]` and references `steps.foo.output` in a param, `foo` appears once in the dependency set. + +### Validation rules + +- `mantle validate` checks for cycles in the dependency graph. +- References to undefined step names are errors. +- `depends_on` must be an array of step names. +- Conditional steps (`if:`) can be depended on — if skipped, dependents still proceed (skipped counts as "resolved"). + +### Cascading failure model + +When a step fails (after exhausting retries): + +- **All steps that transitively depend on the failed step** are marked `cancelled` (not executed). The orchestrator walks the DAG forward from the failed step and sets all reachable pending steps to `cancelled`. +- **Steps with no dependency on the failed step** continue executing. The execution completes in `failed` state once all non-cancelled steps reach a terminal state. +- **`continue_on_error` remains supported where configured.** By default, a failure poisons its downstream subgraph (transitive dependents are cancelled). If a step is configured with `continue_on_error`, that failed step is treated as resolved for dependency scheduling, matching current orchestrator behavior. There is no separate `continue_on_failure` setting — V1.1's cascading failure model applies only to steps without `continue_on_error`. + +## 3. Tool Use Execution Model + +### Workflow YAML syntax + +```yaml +steps: + - name: research_agent + action: ai/completion + params: + model: gpt-4o + system_prompt: "You are a research assistant." + prompt: "Find the current weather in {{ inputs.city }}" + max_tool_rounds: 10 + max_tool_calls_per_round: 5 + tools: + - name: get_weather + description: "Get current weather for a city" + input_schema: + type: object + properties: + city: { type: string, description: "City name" } + required: [city] + action: http/request + params: + url: "https://api.weather.com/v1/current?city={{ tool_input.city }}" + - name: get_forecast + description: "Get weather forecast for a city" + input_schema: + type: object + properties: + city: { type: string, description: "City name" } + required: [city] + action: http/request + params: + url: "https://api.weather.com/v1/forecast?city={{ tool_input.city }}" +``` + +Tools are declared inline on the AI step. Each tool maps to a connector action. `tool_input` is a CEL variable populated from the LLM's tool call arguments. `max_tool_rounds` limits the number of LLM↔tool round-trips (default: 10). `max_tool_calls_per_round` limits tool calls per LLM response (default: 10). + +### Tool input schema + +LLM tool-use APIs (e.g., OpenAI function calling) require a JSON Schema describing each tool's input parameters. Each tool declaration includes an explicit `input_schema`: + +```yaml +tools: + - name: get_weather + description: "Get current weather for a city" + input_schema: + type: object + properties: + city: + type: string + description: "City name" + required: [city] + action: http/request + params: + url: "https://api.weather.com/v1/current?city={{ tool_input.city }}" +``` + +- `description` (required) — human-readable description sent to the LLM. +- `input_schema` (required) — JSON Schema object describing the tool's parameters. Sent directly to the LLM as the function's `parameters` field. +- The LLM's tool call arguments are validated against `input_schema` before execution. Invalid arguments fail the tool call (not the whole step) and the error is returned to the LLM. +- `tool_input` in CEL expressions is populated from the validated arguments object. + +### Execution flow + +The AI connector drives a loop: + +1. **Initial LLM call** — send prompt + tool definitions to the LLM. +2. **LLM responds** — either with final text (done) or with tool call requests. +3. **Tool execution** — for each tool call, the AI connector creates a child `step_execution` row with `parent_step_id` pointing to the AI step. These go through the normal claim/execute/complete flow from Phase 7. +4. **Tool results returned to LLM** — the AI connector collects completed tool results, appends them to the conversation, and calls the LLM again. +5. **Repeat** until the LLM returns a final response or `max_tool_rounds` is exhausted. + +### Sub-step naming and schema + +Child step executions use the `parent_step_id` column from Phase 7: + +```text +step_executions: + id: uuid-child-1 + execution_id: + parent_step_id: uuid-parent-ai-step + step_name: "research_agent/tool/get_weather/0" -- parent/tool/name/round + status: pending → running → completed + output: { tool result } +``` + +The naming convention `parent/tool/name/round` gives unique identification and readable `mantle logs` output. + +### Tool loop orchestration + +The **worker that claimed the AI step** drives the tool-use loop — not the execution-level orchestrator. The AI connector acts as a mini-orchestrator for its sub-steps: + +1. Creates child `step_execution` rows with `status = 'pending'`. +2. Waits for them to be claimed and completed (by any worker, including itself). +3. Collects results and continues the LLM conversation. + +The parent AI step's lease is renewed throughout this process. If the worker crashes, the reaper reclaims the parent step, and recovery kicks in (Section 4). + +## 4. Crash Recovery & Tool Call Caching + +LLMs are non-deterministic — replaying a crashed AI step won't produce the same tool calls. Recovery must be deterministic. + +### What gets cached + +Every LLM response and every tool result is persisted before acting on it. + +**LLM response caching**: After each LLM call, the full response (including tool call requests) is appended to a JSONB array on the parent AI step: + +```sql +UPDATE step_executions + SET cached_llm_responses = cached_llm_responses || $2::jsonb + WHERE id = $1; +``` + +**Tool result caching**: Already handled — each tool call is a child `step_execution` row with its output persisted on completion. + +### Schema addition + +Both `cached_llm_responses` and `parent_step_id` are included in Phase 7's migration (see [Phase 7 Design, Section 2](2026-03-19-multi-node-distribution-design.md#2-schema-changes)). No additional schema migration is needed for Phase 8. + +### Recovery flow + +When a crashed AI step is reclaimed and restarted: + +1. Load `cached_llm_responses` from the parent row. +2. Load all child `step_execution` rows, ordered by `step_name` (which encodes the round index). +3. Reconstruct the conversation using the following algorithm: + +```text +conversation = [system_prompt, user_prompt] +for i, llm_response in enumerate(cached_llm_responses): + append llm_response to conversation + if llm_response contains tool_calls: + for each tool_call in llm_response.tool_calls: + child_name = "{parent}/tool/{tool_call.name}/{i}" + child_row = lookup child step_execution by step_name = child_name + if child_row exists and status == 'completed': + append tool_result(child_row.output) to conversation + elif child_row exists and status in ('pending', 'running'): + wait for child_row to complete, then append + elif child_row does not exist: + create child_row from cached tool_call, wait for completion, append +``` + +4. After replaying all cached rounds: + - If the last cached round ended with all tool results collected: call the LLM with the full reconstructed conversation. + - If the last cached round's tools are still in-flight: wait, then call the LLM. + - If no cached rounds exist (crash before first LLM response): start fresh — call the LLM with the original prompt. + +**Key invariant**: The conversation is always reconstructed from `cached_llm_responses` (interleaved with child row outputs by round), never from memory. This ensures deterministic recovery regardless of which worker picks up the step. + +**Attempt reuse**: Recovery of AI steps reuses the same attempt number. The reaper marks the parent row as `failed`, and the orchestrator creates a new attempt. The new attempt starts with an empty `cached_llm_responses` — it does NOT inherit from the previous attempt. Each attempt is independent. Child rows from the failed attempt remain in the database for auditability but are not referenced by the new attempt. + +### Recovery scenarios + +| Crash point | What's cached | Recovery action | +|---|---|---| +| After LLM response cached, before tool rows created | LLM response | Create tool rows from cached response | +| After tool rows created, tools still running | LLM response + partial tool results | Wait for in-flight tools, create missing ones | +| After all tools complete, before next LLM call | LLM response + all tool results | Call LLM with full conversation history | +| Before LLM response cached | Previous rounds only | Re-call LLM for current round | + +### Idempotency of tool row creation + +Tool child rows use `INSERT ... ON CONFLICT DO NOTHING` keyed on `(execution_id, step_name, attempt)`. The step name `research_agent/tool/get_weather/0` is deterministic from the cached LLM response, so re-creating them after a crash doesn't produce duplicates. + +### Cost of recovery + +Worst case: one redundant LLM call (the one that wasn't cached). All tool executions are either reused from cache or already in-flight. + +## 5. Recursion Limits, Validation & Observability + +### Recursion limits + +Single-level tool use for V1.1. The data model supports future multi-level (child rows can have children), but the AI connector enforces: + +- `max_tool_rounds` — max LLM↔tool round-trips per AI step (default 10, max 50). +- `max_tool_calls_per_round` — max tool calls the LLM can make in a single response (default 10, max 25). +- Total tool executions per AI step capped at `max_tool_rounds × max_tool_calls_per_round` (default 100, max 1250). + +When any limit is hit, the AI connector sends a final message to the LLM: "Tool use limit reached. Provide your best response with available information." If the LLM still returns tool calls, the step fails with a clear error. + +### Validation rules + +`mantle validate` checks: + +- Tool `name` is unique within a step's tool list. +- Each tool's `action` must reference a valid connector. +- Parameters for each tool are valid for that connector (same validation as regular steps). +- `max_tool_rounds` is within bounds. +- `depends_on` cycle detection includes implicit dependencies from CEL expressions. +- Circular tool references are impossible in V1.1 (tools can't declare sub-tools). + +### `mantle logs` output + +```text +EXECUTION abc123 + ✓ fetch_data completed 1.2s + ✓ research_agent completed 8.4s + ├─ round 1 + │ ├─ get_weather completed 0.8s + │ └─ get_forecast completed 1.1s + ├─ round 2 + │ └─ get_weather completed 0.6s + └─ final response "The weather in..." + ✓ send_report completed 0.3s +``` + +Sub-steps are indented under their parent with round grouping. + +### New Prometheus metrics + +- `mantle_tool_calls_total` (counter, labels: `step`, `tool`, `status`) +- `mantle_tool_rounds_total` (counter, labels: `step`) +- `mantle_tool_round_duration_seconds` (histogram) +- `mantle_llm_cache_hits_total` (counter) — recovery replays from cache +- `mantle_parallel_steps_in_flight` (gauge) — concurrent step executions per workflow + +## 6. Configuration + +New `mantle.yaml` keys: + +```yaml +engine: + default_max_tool_rounds: 10 + default_max_tool_calls_per_round: 10 + ai_step_lease_duration: 300s # overrides Phase 7's default step_lease_duration for AI steps +``` diff --git a/docs/superpowers/specs/2026-03-19-multi-node-distribution-design.md b/docs/superpowers/specs/2026-03-19-multi-node-distribution-design.md new file mode 100644 index 0000000..12795e4 --- /dev/null +++ b/docs/superpowers/specs/2026-03-19-multi-node-distribution-design.md @@ -0,0 +1,273 @@ +# Design: Multi-Node Distribution (Phase 7) + +> Phase 7 of Mantle V1.1. Enables multiple Mantle replicas to distribute workflow execution across nodes without duplication or loss. +> +> Designed for medium scale (5–20 replicas, hundreds of concurrent executions). Scaling considerations for 50+ replicas documented in Section 6. + +## 1. Design Decisions + +- **Postgres SKIP LOCKED only** — no external message broker. Preserves the "single binary + Postgres" architecture. +- **Hybrid orchestrator/worker model** — one node orchestrates each execution (determines ready steps), all nodes execute steps. Supports Phase 8's DAG resolution and tool-use sub-steps. +- **Claim and execute in separate transactions** — prevents holding row locks during step execution (which can take seconds to minutes). +- **Step output 1MB hard limit** — JSONB storage with enforced size cap. Users handle large payloads via the S3 connector. Threshold-based offload is a documented future upgrade path. + +## 2. Schema Changes + +### Additions to `step_executions` + +```sql +ALTER TABLE step_executions ADD COLUMN claimed_by TEXT; +ALTER TABLE step_executions ADD COLUMN lease_expires_at TIMESTAMPTZ; +ALTER TABLE step_executions ADD COLUMN max_attempts INTEGER NOT NULL DEFAULT 1; +ALTER TABLE step_executions ADD COLUMN parent_step_id UUID REFERENCES step_executions(id); +ALTER TABLE step_executions ADD COLUMN cached_llm_responses JSONB DEFAULT '[]'::jsonb; + +CREATE INDEX idx_step_executions_claimable + ON step_executions (execution_id, status) + WHERE status = 'pending'; +``` + +- `claimed_by` — node identifier (`hostname:pid` or UUID), set when a worker claims a step. +- `lease_expires_at` — null when not leased. Workers must renew before expiry or the reaper reclaims the step. +- `max_attempts` — populated from the step's retry policy at row creation time. Denormalized here so the reaper can make retry decisions without loading the workflow definition. +- `parent_step_id` — null for top-level steps. Added now as a forward-looking schema addition so Phase 8 doesn't require an ALTER TABLE on a hot table. No Phase 7 code reads or writes this column. +- `cached_llm_responses` — used by Phase 8 for tool-use crash recovery. Added in the same migration to avoid future schema changes. + +All columns added in a single goose migration (next sequential number after the latest existing migration). The migration includes the `execution_claims` table creation. + +### New table: `execution_claims` + +```sql +CREATE TABLE execution_claims ( + execution_id UUID PRIMARY KEY REFERENCES workflow_executions(id), + claimed_by TEXT NOT NULL, + lease_expires_at TIMESTAMPTZ NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); +``` + +Separates "who is orchestrating this execution" from "who is running this step." A node claims an execution to orchestrate it (determine which steps are ready, create pending rows). Any node can claim individual steps to execute. + +No new tables for the work queue — `step_executions` with `status = 'pending'` IS the queue. + +## 3. Claim/Execute Transaction Model + +### Claim transaction (fast, milliseconds) + +```sql +BEGIN; +SELECT id, step_name FROM step_executions + WHERE execution_id = $1 AND status = 'pending' + ORDER BY created_at + LIMIT 1 + FOR UPDATE SKIP LOCKED; + +UPDATE step_executions + SET status = 'running', claimed_by = $2, + lease_expires_at = NOW() + $3::interval, + started_at = NOW() + WHERE id = $4; +COMMIT; +``` + +### Execute (outside any transaction) + +Run the connector, get the result. + +### Complete transaction — success (fast) + +```sql +UPDATE step_executions + SET status = 'completed', output = $2, + completed_at = NOW(), lease_expires_at = NULL + WHERE id = $1 AND claimed_by = $3 AND status = 'running'; +``` + +### Complete transaction — failure (fast) + +```sql +UPDATE step_executions + SET status = 'failed', error = $2, + completed_at = NOW(), lease_expires_at = NULL + WHERE id = $1 AND claimed_by = $3 AND status = 'running'; +``` + +Both completion queries include `AND status = 'running'` to prevent a late-completing worker from overwriting a step that was already reclaimed and transitioned by the reaper. The `claimed_by` check acts as a **fencing token** — if the reaper reclaimed this step and another worker picked it up, the original worker's completion silently fails (0 rows updated). The worker detects this (0 rows affected) and discards its result. + +**Fencing race note**: If a worker's lease expires but it completes work before another worker claims the step, the completion returns 0 rows (the reaper cleared `claimed_by`). This is safe — the step is back in `pending` and another worker will pick it up. No work is lost, though the original result is discarded. + +### Lease renewal + +For long-running steps, workers periodically extend their lease: + +```sql +UPDATE step_executions + SET lease_expires_at = NOW() + $2::interval + WHERE id = $1 AND claimed_by = $3 AND status = 'running'; +``` + +If this returns 0 rows, the lease was already reclaimed — the worker aborts execution via context cancellation. + +## 4. Orchestrator Loop & Worker Loop + +Each replica runs two goroutines concurrently. + +### Orchestrator loop + +Polls for unclaimed executions (new runs or orphaned orchestrations): + +1. Claim an execution via `execution_claims` using `INSERT ... ON CONFLICT DO NOTHING` (only one node wins). +2. Load the workflow definition and completed steps. +3. Evaluate the step DAG — find steps whose dependencies are all satisfied. +4. INSERT pending `step_executions` rows for ready steps. +5. Wait for those steps to complete (poll `step_executions` for status changes). +6. Repeat steps 3–5 until all steps are done or a step fails. +7. Update `workflow_executions.status` to `completed` or `failed`. +8. Delete the `execution_claims` row. + +The orchestrator renews its lease on `execution_claims` periodically. If it crashes, the reaper releases the claim, and another node's orchestrator picks it up — loading completed steps from the checkpoint and continuing. + +**Poll interval**: 500ms for step completion checks. Jittered ±100ms. + +### Worker loop + +Polls for claimable steps across all executions: + +1. `SELECT ... FOR UPDATE SKIP LOCKED` from `step_executions WHERE status = 'pending'`. +2. Claim, execute, complete (per Section 3's transaction model). +3. Loop. + +**Poll interval**: 200ms base with exponential backoff up to 5s when no work is found. Resets to 200ms on successful claim. Jittered ±50ms to mitigate thundering herd. + +### Thundering herd mitigation + +Jitter plus SKIP LOCKED means even 20 replicas polling simultaneously get different rows — no contention. Backoff when idle keeps Postgres load low during quiet periods. + +## 5. Reaper + +A single goroutine per replica reclaims orphaned work. All replicas run the reaper — SKIP LOCKED prevents conflicts between them. + +### Step reaper (runs every 30s) + +**Retry model**: When the reaper reclaims a step, it marks the current row as `failed` and the orchestrator is responsible for creating a new `step_execution` row with `attempt + 1` (matching the existing unique constraint `(execution_id, step_name, attempt)`). This is consistent with the existing retry model where each attempt is a separate row. + +Mark expired steps as failed: + +```sql +UPDATE step_executions + SET status = 'failed', error = 'lease expired', + completed_at = NOW(), claimed_by = NULL, lease_expires_at = NULL + WHERE status = 'running' + AND lease_expires_at < NOW() + RETURNING id, step_name, execution_id, attempt, max_attempts; +``` + +The orchestrator, on its next poll, checks for failed steps where `attempt < max_attempts` and creates a new row with `attempt + 1`. Steps where `attempt >= max_attempts` remain failed and trigger execution failure handling. + +### Execution orchestrator reaper (runs every 30s, offset 15s from step reaper) + +```sql +DELETE FROM execution_claims + WHERE lease_expires_at < NOW() + RETURNING execution_id; +``` + +Released executions are picked up by another node's orchestrator loop on its next poll. + +### Lease durations + +| Work type | Default lease | Renewal interval | +|---|---|---| +| Step execution | 60s | Every 20s | +| Execution orchestration | 120s | Every 40s | +| AI/LLM steps | 300s | Every 60s | + +Lease duration is configurable per step type via `timeout` in workflow YAML — lease is set to `timeout + 30s` buffer. The defaults above apply when no timeout is specified. + +### Reaper consistency + +Multiple reapers running is safe — the UPDATE/DELETE statements are idempotent. If two reapers fire simultaneously, one gets the rows and the other gets 0 rows affected. + +### Orchestrator failure handling + +When a step fails (either from connector error or reaper expiry), the orchestrator: + +1. Checks if `attempt < max_attempts` — if so, creates a new `step_execution` row with `attempt + 1` and `status = 'pending'`. +2. If retries are exhausted, marks the `workflow_executions` row as `failed`. All pending steps for this execution are set to `cancelled` (not executed). +3. In-flight steps for the same execution are allowed to complete (their results are checkpointed but the execution is already marked failed). + +### Timestamp discipline + +All lease timestamps use database server time via `NOW()`, never application-level timestamps. This eliminates clock skew issues between replicas and the database. Workers and reapers never compute lease expiry using local clocks. + +## 6. Scaling Considerations + +### Medium scale (5–20 replicas) — design target + +- SKIP LOCKED contention is negligible at this scale. +- Connection count: ~4 connections per replica (orchestrator, worker, reaper, lease renewal). 20 replicas = 80 connections, within Postgres defaults. +- Partial index on `status = 'pending'` keeps claim queries fast regardless of historical row count. + +### Large scale inflection points (50+ replicas) + +**Connection pooling**: At 50+ replicas (150+ connections), deploy PgBouncer in transaction mode. The claim/execute/complete pattern uses short transactions, so this works out of the box. + +**Poll frequency tuning**: 50+ workers polling at 200ms = 250 queries/second when idle. Exponential backoff to 5s reduces this to ~10 queries/second at idle. If bursts are a concern, `LISTEN/NOTIFY` can replace polling — the orchestrator does `NOTIFY step_ready` after inserting pending rows, workers `LISTEN` instead of polling. Noted as a future optimization. + +**Completed row archival**: Add a `completed_before` archival job that moves rows older than a configurable retention period (default 30 days) to `step_executions_archive`. The partial index on `status = 'pending'` means query performance is unaffected by table size, but storage and vacuum costs grow. + +**Step output size limit**: 1MB hard limit enforced at the engine level before writing to JSONB. Steps exceeding this fail with a clear error suggesting the S3 connector pattern. Threshold-based offload to object storage is the natural upgrade path. + +## 7. Observability + +### New Prometheus metrics + +- `mantle_queue_depth` (gauge) — count of pending steps +- `mantle_claim_duration_seconds` (histogram) — time from pending to claimed +- `mantle_lease_renewals_total` (counter) +- `mantle_lease_expirations_total` (counter) — indicates node failures or slow steps +- `mantle_reaper_reclaimed_total` (counter) + +## 8. Testing Strategy + +Concurrency tests are required to prove correctness under load with deliberate crashes. + +### Test harness + +Integration tests using testcontainers (real Postgres). Multiple worker goroutines within a single test process simulate multi-node behavior — each goroutine gets its own `node_id` and runs independent orchestrator/worker/reaper loops. + +### Invariants to assert + +- **No step lost**: every step in every workflow execution reaches a terminal state (`completed`, `failed`, `skipped`, `cancelled`). +- **No step duplicated**: for a given `(execution_id, step_name, attempt)`, at most one row has `status = 'completed'` with non-null output. +- **All executions complete**: every workflow execution reaches `completed` or `failed` within a timeout. +- **Fencing correctness**: a worker whose lease expired cannot overwrite a step completed by another worker. + +### Crash simulation tests + +1. **Worker crash mid-execution**: Kill a worker goroutine (cancel its context) while a step is `running`. Verify: reaper reclaims the step within `lease_timeout + reaper_interval`, another worker picks it up, step completes. +2. **Orchestrator crash**: Kill the orchestrator goroutine for an execution. Verify: reaper releases the `execution_claims` row, another orchestrator picks up the execution and continues from checkpoint. +3. **Multiple simultaneous crashes**: Kill 2 of 3 workers. Verify: remaining worker completes all work, no steps lost. +4. **Reaper under load**: Run 10 workflows with 5 workers, kill 3 workers mid-flight. Verify: all workflows eventually complete with the remaining 2 workers. + +### Load tests + +- 3+ worker goroutines, 50 concurrent workflow executions, each with 5-10 steps. +- Assert all invariants hold after completion. +- Measure: claim latency, queue depth over time, total throughput. + +## 9. Configuration + +New `mantle.yaml` keys under `engine:`: + +```yaml +engine: + node_id: "" # auto-generated if empty (hostname:pid) + worker_poll_interval: 200ms + worker_max_backoff: 5s + orchestrator_poll_interval: 500ms + step_lease_duration: 60s + orchestration_lease_duration: 120s + reaper_interval: 30s + step_output_max_bytes: 1048576 # 1MB +``` diff --git a/docs/superpowers/specs/2026-03-24-data-transformation-design.md b/docs/superpowers/specs/2026-03-24-data-transformation-design.md new file mode 100644 index 0000000..15197f7 --- /dev/null +++ b/docs/superpowers/specs/2026-03-24-data-transformation-design.md @@ -0,0 +1,175 @@ +# Data Transformation — CEL Functions & Documentation + +**Date:** 2026-03-24 +**Issue:** [#14 — Data Transformation Step](https://github.com/dvflw/mantle/issues/14) +**Status:** Draft + +## Problem + +Mantle workflows can pass data between steps via CEL expressions, but lack the tools to reshape that data. The common pattern — fetch from API, normalize for a DB schema, store — requires either manual field-by-field construction (not possible in CEL today) or routing through the AI connector (slow, expensive, non-deterministic for structural transforms). + +## Discovery: Existing Hidden Capabilities + +CEL's default environment includes macros that already work in Mantle but were never documented or tested: + +- `.map(item, expr)` — transform each element in a list +- `.filter(item, expr)` — keep elements matching a predicate +- `.exists(item, expr)` — true if any element matches +- `.all(item, expr)` — true if all elements match +- `.exists_one(item, expr)` — true if exactly one matches + +These need documentation and tests, not implementation. + +## Design + +### Custom CEL Functions + +All functions registered in `internal/cel/functions.go` via `cel.Function()` options passed to `cel.NewEnv()`. Pure functions, no side effects. + +#### String Functions (methods on string type) + +| Function | Example | Result | +|----------|---------|--------| +| `toLower()` | `"HELLO".toLower()` | `"hello"` | +| `toUpper()` | `"hello".toUpper()` | `"HELLO"` | +| `trim()` | `" hello ".trim()` | `"hello"` | +| `replace(old, new)` | `"foo-bar".replace("-", "_")` | `"foo_bar"` | +| `split(delim)` | `"a,b,c".split(",")` | `["a", "b", "c"]` | + +#### Type Coercion (global functions) + +| Function | Example | Result | +|----------|---------|--------| +| `parseInt(string)` | `parseInt("42")` | `42` | +| `parseFloat(string)` | `parseFloat("3.14")` | `3.14` | +| `toString(any)` | `toString(42)` | `"42"` | + +#### Object Construction (global function) + +| Function | Example | Result | +|----------|---------|--------| +| `obj(k1, v1, k2, v2, ...)` | `obj("name", "alice", "age", 30)` | `{"name": "alice", "age": 30}` | + +Errors on odd number of args or non-string keys. Enables building maps for DB inserts and API payloads. + +#### Null Coalescing (global function) + +| Function | Example | Result | +|----------|---------|--------| +| `default(value, fallback)` | `default(steps.x.output.json.name, "unknown")` | value if non-null, else `"unknown"` | + +#### JSON (global functions) + +| Function | Example | Result | +|----------|---------|--------| +| `jsonEncode(value)` | `jsonEncode(obj("a", 1))` | `'{"a":1}'` | +| `jsonDecode(string)` | `jsonDecode('{"a":1}')` | `{"a": 1}` | + +#### Date/Time (global functions) + +| Function | Example | Result | +|----------|---------|--------| +| `parseTimestamp(string)` | `parseTimestamp("2026-03-24T19:00:00Z")` | timestamp value | +| `formatTimestamp(ts, layout)` | `formatTimestamp(ts, "2006-01-02")` | `"2026-03-24"` | + +Uses Go time layout strings. + +#### Collections (global function) + +| Function | Example | Result | +|----------|---------|--------| +| `flatten(list)` | `flatten([[1,2],[3,4]])` | `[1,2,3,4]` | + +### Integration Point + +In `internal/cel/cel.go`, the `NewEvaluator` function passes function options to `cel.NewEnv()`: + +```go +func NewEvaluator() (*Evaluator, error) { + opts := []cel.EnvOption{ + cel.Variable("steps", cel.MapType(cel.StringType, cel.DynType)), + cel.Variable("inputs", cel.MapType(cel.StringType, cel.DynType)), + cel.Variable("env", cel.MapType(cel.StringType, cel.StringType)), + cel.Variable("trigger", cel.MapType(cel.StringType, cel.DynType)), + } + opts = append(opts, customFunctions()...) + + env, err := cel.NewEnv(opts...) + // ... +} +``` + +`customFunctions()` is defined in `functions.go` and returns `[]cel.EnvOption`. + +### Error Handling + +All errors surface through the existing `Eval` error path: +- Type mismatches: `parseInt("abc")` → evaluation error +- `obj()` with odd args → evaluation error +- `obj()` with non-string keys → evaluation error +- `jsonDecode()` with invalid JSON → evaluation error +- `parseTimestamp()` with unparseable string → evaluation error + +No new error types needed. + +## Documentation + +### CEL Expressions Reference Update + +Update `site/src/content/docs/concepts/expressions.md` to add: +- All custom functions organized by category +- The already-working macros (`.map()`, `.filter()`, `.exists()`, `.all()`, `.exists_one()`) +- Examples for each function + +### New: Data Transformations Guide + +New page at `site/src/content/docs/getting-started/data-transformations.md` covering three patterns: + +**Pattern 1 — Structural transforms (CEL only):** +API result → `.map()` + `obj()` → Postgres INSERT. No AI needed. For when the transform is a known schema mapping. + +**Pattern 2 — AI-powered transforms:** +Unstructured data → AI connector with `output_schema` → structured output. For when the transform requires interpretation, classification, or natural language understanding. + +**Pattern 3 — Hybrid:** +Fetch → CEL for structural normalization → AI for enrichment/classification → Store. Combines both approaches. + +Each pattern includes a complete example workflow YAML. + +### New Example Workflows + +- `examples/data-transform-api-to-db.yaml` — Fetch API → CEL `.map()` + `obj()` → Postgres INSERT (the exact use case from the issue) +- `examples/ai-data-enrichment.yaml` — Fetch data → AI classify/enrich with structured output → store + +## Files Changed + +### Modified + +| File | Change | +|------|--------| +| `internal/cel/cel.go` | Pass `customFunctions()` options to `cel.NewEnv()` | +| `site/src/content/docs/concepts/expressions.md` | Add function reference, document macros | + +### New + +| File | Purpose | +|------|---------| +| `internal/cel/functions.go` | All custom function definitions | +| `internal/cel/functions_test.go` | Table-driven tests for every custom function | +| `internal/cel/macros_test.go` | Tests for built-in macros (lock in existing behavior) | +| `site/src/content/docs/getting-started/data-transformations.md` | Transformation patterns guide | +| `examples/data-transform-api-to-db.yaml` | Structural transform example workflow | +| `examples/ai-data-enrichment.yaml` | AI transform example workflow | + +## Non-Goals + +- **Custom user-defined functions** — no plugin/extension API for CEL functions +- **Loops or control flow** — CEL is intentionally non-Turing-complete +- **Regex** — deferring to a future issue; CEL's `matches()` function could be enabled later +- **New connector type** — transformations happen in CEL expressions, not as a separate step type + +## Testing Strategy + +- **`functions_test.go`** — table-driven: each function gets happy path + error cases (wrong types, empty inputs, edge cases) +- **`macros_test.go`** — tests for `.map()`, `.filter()`, `.exists()`, `.all()`, `.exists_one()` with list data to lock in behavior +- **Existing tests unaffected** — custom functions are additive; no behavior changes to existing expressions diff --git a/docs/superpowers/specs/2026-03-25-docker-connector-artifacts-design.md b/docs/superpowers/specs/2026-03-25-docker-connector-artifacts-design.md new file mode 100644 index 0000000..51ea761 --- /dev/null +++ b/docs/superpowers/specs/2026-03-25-docker-connector-artifacts-design.md @@ -0,0 +1,381 @@ +# Docker Connector & Artifact System Design + +## Overview + +Add a Docker connector (`docker/run`) for running containers to completion as workflow steps, and an artifact system for passing large files between steps without loading them into engine memory. Covers GitHub issues #12 (Docker Volume Backup) and #13 (Run Docker Container for Step). + +## Docker Connector + +### Action: `docker/run` + +Runs a container to completion, captures exit code, stdout, and stderr. No long-running container management — run-to-completion only. + +Uses the official Moby Go client (`github.com/docker/docker/client`) to talk to the Docker daemon via socket or TCP. + +### Params + +| Param | Type | Required | Default | Description | +|-------|------|----------|---------|-------------| +| `image` | string | yes | — | Container image | +| `cmd` | []string | no | — | Command and arguments | +| `env` | map[string]string | no | — | Environment variables | +| `stdin` | string | no | — | Data piped to container stdin | +| `mounts` | []mount | no | — | Volume/bind mounts | +| `network` | string | no | `"bridge"` | Docker network mode | +| `pull` | string | no | `"missing"` | Pull policy: `always`, `missing`, `never` | +| `memory` | string | no | — | Memory limit (e.g., `"512m"`) | +| `cpus` | float | no | — | CPU limit | +| `remove` | bool | no | `true` | Remove container on completion | + +Each mount has: `source` (volume name or host path), `target` (container path), `readonly` (bool, optional). + +### Output + +```json +{ + "exit_code": 0, + "stdout": "...", + "stderr": "..." +} +``` + +stdout and stderr are capped at 10MB (`DefaultMaxResponseBytes`). No truncation flag — this limit is documented. The connector logs a warning when truncation occurs. + +### Exit Code Semantics + +A non-zero exit code does NOT constitute a step failure. The step always succeeds (unless the Docker API itself errors), and the exit code is informational in the output. Workflow authors use `if` conditions to branch on exit code. This allows subsequent steps (e.g., failure notifications) to execute regardless of the container's exit code. + +### Credentials + +**Docker daemon** — new `docker` credential type: + +| Field | Required | Description | +|-------|----------|-------------| +| `host` | no | Defaults to `unix:///var/run/docker.sock` | +| `ca_cert` | no | TLS CA certificate | +| `client_cert` | no | TLS client certificate | +| `client_key` | no | TLS client key | + +All fields optional. An empty credential means "local socket, no TLS." For remote TCP with TLS, populate all four. + +**Registry auth** — uses existing `basic` credential type via `registry_credential` param on the step. Tokens go in the password field (this is the standard Docker registry API pattern). + +```yaml +- name: run-private-image + action: docker/run + credential: my-docker + registry_credential: my-registry + params: + image: myorg/processor:latest + cmd: ["process"] +``` + +### Container Lifecycle + +1. Pull image (per pull policy; authenticate with `registry_credential` if provided) +2. Create container with params (env, cmd, mounts, network, resource limits) +3. If `stdin` is set, attach and pipe it +4. If step declares `artifacts`, mount scratch dir at `/mantle/artifacts` +5. Start container, wait for exit +6. Capture stdout, stderr, exit code +7. If step declares `artifacts`, persist declared files to tmp storage (fail step if declared artifact path is missing) +8. Remove container (if `remove: true`) + +### Timeout and Kill Chain + +The step-level `timeout` controls the overall execution time. When the context is cancelled (timeout or workflow cancellation): + +1. Connector calls `ContainerStop` with a 10-second grace period (SIGTERM) +2. If the container does not exit within the grace period, Docker sends SIGKILL +3. stdout/stderr captured up to the point of termination +4. Step output includes whatever was captured; exit code reflects the signal + +## Artifact System + +### Purpose + +Allow steps to produce large files that subsequent steps can reference without loading into engine memory. Any connector can produce artifacts, not just Docker. + +### Declaration + +Workflow authors explicitly declare artifacts on a step: + +```yaml +steps: + - name: backup-volume + action: docker/run + params: + image: alpine + cmd: ["tar", "-czf", "/mantle/artifacts/backup.tar.gz", "-C", "/data", "."] + mounts: + - source: "my-volume" + target: "/data" + readonly: true + artifacts: + - path: /mantle/artifacts/backup.tar.gz + name: backup-archive +``` + +### Engine Behavior + +1. Step declares `artifacts` → engine creates a scratch dir, passes the path via context (using `WithArtifactsDir(ctx, path)`, following the established `WithExecutionID` pattern) +2. For `docker/run`: connector reads the artifacts dir from context and mounts it at `/mantle/artifacts` inside the container +3. For other connectors: connector reads the artifacts dir from context and writes files there +4. After step completes, engine persists declared files from scratch dir to tmp storage. If a declared artifact path is missing from the scratch dir, the step fails with an error. +5. Metadata written to `execution_artifacts` table +6. Artifacts cleaned up after retention TTL expires + +### CEL Access + +``` +artifacts['backup-archive'].url # S3 URI or local path +artifacts['backup-archive'].size # bytes +artifacts['backup-archive'].name # "backup-archive" +``` + +### Validation + +`mantle validate` rejects duplicate artifact names across steps within a workflow. Artifact names must be unique per workflow definition. + +### Artifact Access Scoping + +Artifacts are available to any step that executes after the producing step completes. If a step references an artifact from a step that was skipped (via `if` condition), this is a runtime error. The engine checks artifact availability before executing a step that references artifacts in its params. + +### Template Syntax Note + +In the example workflows, `{{ }}` in `params` values denotes string interpolation (CEL expressions embedded in strings). The `if` field uses bare CEL expressions without delimiters. This is the existing convention in the codebase. + +### Artifact-Aware Connectors + +Connectors that consume artifacts (e.g., `s3/put`) need to detect when a param value is an artifact URL and stream from it instead of treating it as literal content. This avoids loading large files into memory. + +## Database Schema + +### `execution_artifacts` Table + +```sql +CREATE TABLE execution_artifacts ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + execution_id UUID NOT NULL REFERENCES workflow_executions(id), + step_name TEXT NOT NULL, + name TEXT NOT NULL, + url TEXT NOT NULL, + size BIGINT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + UNIQUE (execution_id, name) +); +``` + +The unique constraint on `(execution_id, name)` enforces artifact name uniqueness at the database level. + +## Tmp Storage Configuration + +### Purpose + +System-level config for where artifacts are persisted between steps. Supports distributed deployments (S3) and local dev (filesystem). + +### Config in `mantle.yaml` + +```yaml +# S3 (production / multi-pod) +tmp: + type: s3 + bucket: mantle-tmp + prefix: tmp/ + retention: 24h + +# Filesystem (local dev / single node) +tmp: + type: filesystem + path: /tmp/mantle-scratch + retention: 24h +``` + +### Namespacing + +Files are stored at: `tmp/{workflow-name}/{execution-id}/{artifact-name}` + +### Retention + +- Default TTL applies to both successful and failed executions +- Configurable: operator can change duration or disable auto-cleanup +- Engine runs a reaper to clean up expired artifacts +- If no tmp storage is configured and a workflow declares artifacts, `mantle validate` warns and `mantle apply` rejects + +## Example Workflows + +### Docker Volume Backup (Issue #12) + +```yaml +name: docker-volume-backup +description: > + Back up a Docker volume to S3 on a daily schedule. + Compresses the volume contents, uploads to S3, and + sends a Slack alert on success or failure. + +triggers: + - type: cron + schedule: "0 2 * * *" + +steps: + - name: backup-volume + action: docker/run + credential: my-docker + timeout: "10m" + params: + image: alpine + cmd: ["tar", "-czf", "/mantle/artifacts/backup.tar.gz", "-C", "/data", "."] + mounts: + - source: "my-app-data" + target: "/data" + readonly: true + memory: "256m" + remove: true + artifacts: + - path: /mantle/artifacts/backup.tar.gz + name: backup-archive + + - name: upload-to-s3 + action: s3/put + credential: aws-prod + timeout: "5m" + params: + bucket: my-backups + key: "volumes/my-app-data/{{ date }}/backup.tar.gz" + content: "{{ artifacts['backup-archive'].url }}" + content_type: "application/gzip" + + - name: notify-success + action: slack/send + credential: slack-token + if: "steps['upload-to-s3'].output.size > 0" + params: + channel: "#ops-alerts" + text: "Volume backup completed — {{ artifacts['backup-archive'].size }} bytes uploaded to s3://my-backups/volumes/my-app-data/" + + - name: notify-failure + action: slack/send + credential: slack-token + if: "steps['upload-to-s3'].error != null" + params: + channel: "#ops-alerts" + text: "Volume backup FAILED — {{ steps['upload-to-s3'].error }}" +``` + +### Data Processing with Docker (Issue #13) + +```yaml +name: docker-data-transform +description: > + Query a database for recent records, process them through + a containerized tool, and post the results to Slack. + Demonstrates the docker/run connector with stdin/stdout + data passing. + +steps: + - name: fetch-records + action: postgres/query + credential: my-db + timeout: "15s" + params: + query: "SELECT id, name, status, updated_at FROM tasks WHERE updated_at > now() - interval '24 hours' ORDER BY updated_at DESC" + + - name: transform-data + action: docker/run + credential: my-docker + timeout: "30s" + params: + image: giantswarm/tiny-tools + cmd: ["jq", "[.[] | {id, name, status}]"] + stdin: "{{ steps['fetch-records'].output.json }}" + memory: "128m" + + - name: post-results + action: slack/send + credential: slack-token + if: "steps['transform-data'].output.exit_code == 0" + params: + channel: "#daily-report" + text: "Daily task summary:\n{{ steps['transform-data'].output.stdout }}" +``` + +## Go Type Changes + +### New fields on `Step` struct (`internal/workflow/workflow.go`) + +```go +type Step struct { + Name string `yaml:"name"` + Action string `yaml:"action"` + Params map[string]any `yaml:"params"` + If string `yaml:"if"` + Retry *RetryPolicy `yaml:"retry"` + Timeout string `yaml:"timeout"` + Credential string `yaml:"credential"` + RegistryCredential string `yaml:"registry_credential"` // NEW: for docker/run private image pulls + DependsOn []string `yaml:"depends_on"` + Artifacts []ArtifactDecl `yaml:"artifacts"` // NEW: artifact declarations +} +``` + +### New types + +```go +// ArtifactDecl declares a file that a step will produce. +type ArtifactDecl struct { + Path string `yaml:"path"` // path inside the artifacts dir + Name string `yaml:"name"` // unique name for CEL reference +} + +// ArtifactRef is the runtime representation available in CEL expressions. +type ArtifactRef struct { + Name string `json:"name"` + URL string `json:"url"` // S3 URI or local filesystem path + Size int64 `json:"size"` +} +``` + +### New context helpers (`internal/engine/`) + +```go +func WithArtifactsDir(ctx context.Context, dir string) context.Context +func ArtifactsDirFromContext(ctx context.Context) string +``` + +## Security Considerations + +Docker socket access (`unix:///var/run/docker.sock`) grants effectively root-level access to the host. For V1: + +- The `docker` credential type controls which daemon a workflow connects to +- Resource limits (`memory`, `cpus`) constrain container resource usage +- The `remove: true` default prevents container accumulation + +Future considerations (out of scope for this design): +- Image allowlists to restrict which images can be pulled +- Preventing `--privileged` mode (not exposed as a param, so blocked by default) +- Mount source restrictions to prevent mounting sensitive host paths +- Network policy enforcement + +## Homepage Update + +Add the Docker connector to the connectors section in `site/src/components/Connectors.astro`. Update the count from "7 connectors with 9 actions" to "8 connectors with 10 actions." + +## Design Decisions + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Docker SDK | Moby Go client | Official, supports socket + TCP + TLS, API version negotiation | +| Container lifecycle | Run-to-completion only | Long-running containers handled via HTTP/plugin connectors | +| Input mechanisms | env, cmd, stdin, mounts | Covers all common data-passing patterns | +| Output mechanisms | exit_code, stdout, stderr | File output handled via artifact system | +| stdout/stderr cap | 10MB | Matches `DefaultMaxResponseBytes`, prevents OOM | +| Artifact declaration | Explicit on step | IaC-first philosophy, self-documenting, no magic | +| Artifact storage | Separate `execution_artifacts` table | Clean querying and cleanup, no output bloat | +| Artifact CEL access | `artifacts['name']` (top-level) | Short, clean; uniqueness enforced at validation | +| Artifact connector interface | Context value (`WithArtifactsDir`) | Follows established `WithExecutionID` pattern, no param pollution | +| Tmp storage | System-level config in `mantle.yaml` | One config for all workflows, S3 or filesystem | +| Tmp namespacing | `tmp/{workflow}/{execution}/{artifact}` | Isolated per execution, easy cleanup | +| Tmp retention | TTL-based, configurable | Useful for both debugging and auditing | +| Artifact scope | Single execution | Clean lifecycle, no cross-execution dependencies | +| Registry auth | Existing `basic` credential type | Industry-standard pattern, tokens go in password field | +| Large file transfer | Artifact URLs, not content | Prevents loading multi-GB files into engine memory | diff --git a/docs/superpowers/specs/2026-03-26-v040-safety-net-design.md b/docs/superpowers/specs/2026-03-26-v040-safety-net-design.md new file mode 100644 index 0000000..f9deb73 --- /dev/null +++ b/docs/superpowers/specs/2026-03-26-v040-safety-net-design.md @@ -0,0 +1,488 @@ +# v0.4.0 — The Safety Net Update: Design Spec + +**Milestone:** [v0.4.0](https://github.com/dvflw/mantle/milestone/5) +**Issues:** #52, #75, #51, #49, #30, #48, #50 (7 issues; #70 release pipeline is a separate plan) +**Implementation order:** #52 → #75 → #51 → #49 → #30 → #48 → #50 + +--- + +## 1. Config File Versioning (#52) + +Add a top-level `version` field to `mantle.yaml` as a guard rail for future config changes. + +### Schema + +```yaml +version: 1 +database: + url: postgres://... +``` + +### Behavior on Load + +| `version` value | Behavior | +|-----------------|----------| +| Missing or `0` | Treat as `1`, no warning (backward compat for existing installs) | +| `1` | Valid, proceed | +| `2+` | Hard error: `"unsupported config version %d; this version of mantle supports config version 1 — upgrade mantle or check your mantle.yaml"` | + +### Changes + +- `internal/config/config.go`: Add `Version int` to `Config`, validate in `Load()`. No env var binding — `version` is config-file-only (setting it via `MANTLE_VERSION` would be confusing and error-prone). +- Docs: Update `configuration.md`, update all example configs + +### What This Does NOT Do + +No migration logic. Version 1 is the only version. The field exists so future config changes can bump the version and old binaries reject new configs cleanly rather than silently zero-valuing new fields. + +--- + +## 2. Rename tmp Config Section to storage (#75) + +The `tmp` config section stores execution artifacts (screenshots, PDFs, reports) with configurable retention. These are not temporary files — the naming is misleading. + +### Schema Change + +```yaml +# Before +tmp: + type: s3 + bucket: mantle-artifacts + prefix: artifacts/ + retention: 24h + +# After +storage: + type: s3 + bucket: mantle-artifacts + prefix: artifacts/ + retention: 24h +``` + +Environment variables: `MANTLE_TMP_*` → `MANTLE_STORAGE_*` + +### Backward Compatibility + +Within config version 1: if `storage` is not set but `tmp` is present, emit a deprecation warning via `slog.Warn` and copy the values over. The `tmp` fallback gets removed in a future version bump. + +### Changes + +- `internal/config/config.go`: Rename `TmpConfig` → `StorageConfig`, `Tmp` → `Storage`, add `tmp` fallback logic with deprecation warning +- `internal/artifact/tmp.go` → `internal/artifact/storage.go`: Rename `TmpStorage` interface → `Storage`, `FilesystemTmpStorage` struct → `FilesystemStorage` +- `internal/engine/engine.go`: Update `TmpStorage` field name +- `internal/cli/`, `internal/server/`: Update references +- `docker-compose.yml`: Update env vars +- Docs: `configuration.md`, `deployment-guide.md` + +--- + +## 3. Secret Key Rotation (#51) + +New CLI command to re-encrypt all credentials with a new AES-256-GCM master key. + +### Command + +``` +mantle secrets rotate-key [--new-key ] +``` + +- `--new-key` is **optional**. If omitted, a new 32-byte key is auto-generated and printed to stdout. +- Also accepts `MANTLE_NEW_ENCRYPTION_KEY` env var. +- **Requires admin privileges** (admin API key or admin OIDC user). Non-admin callers get a permission error. + +### Flow + +1. Verify caller has admin privileges +2. Load current encryption key from config (`encryption.key`) +3. If `--new-key` provided, use it; otherwise generate via `secret.GenerateKey()` and print it +4. Create two `secret.Encryptor` instances (old key, new key) +5. Open a single transaction +6. `SELECT id, name, encrypted_data, nonce FROM credentials` — **all teams, global scope** +7. For each credential: decrypt with old encryptor → re-encrypt with new encryptor (fresh random 12-byte GCM nonce) +8. `UPDATE` each row in place +9. Commit transaction (all-or-nothing — if any credential fails to decrypt, abort and report which credential failed, including both `name` and `id` in the error message) +10. Emit `ActionSecretKeyRotated` audit event with credential count +11. Print: `"Rotated N credentials. Update your mantle.yaml encryption.key and restart."` + +### Design Decisions + +- **Single transaction:** If any credential fails to decrypt, the whole rotation rolls back. No partial state. The error identifies the specific failed credential. +- **Global scope:** Rotation operates across all teams. The master key is system-wide, not per-team. +- **No automatic config rewrite:** The command tells the user to update their config. Writing config files is error-prone (comments stripped, formatting changes). +- **Algorithm unchanged:** Rotation changes the key, not the cipher. AES-256-GCM throughout. + +### Changes + +- `internal/cli/secrets_rotate_key.go`: New command under `secrets` subcommand. CLI owns the transaction: begins tx, calls `RotateAll`, commits or rolls back. +- `internal/secret/store.go`: Add `RotateAll(ctx, tx, oldEncryptor, newEncryptor) (int, error)`. This **replaces** the existing `ReEncryptAll` method, removing its `WHERE team_id = $1` filter to operate globally. The caller provides the transaction. Error messages include both credential name and ID. +- `internal/audit/actions.go`: Add `ActionSecretKeyRotated` + +--- + +## 4. Concurrency Controls (#49) + +Three concurrency dimensions to prevent execution floods. + +### Concurrency Dimensions + +| Dimension | Config Location | Default | +|-----------|----------------|---------| +| Per-workflow `max_parallel_executions` | Workflow YAML | `0` (unlimited) | +| Per-team global limit | `mantle.yaml` default + per-team DB override | `0` (unlimited) | +| Per-step `max_parallel` | Step YAML (fan-out steps) | `0` (unlimited) | + +### YAML Schema + +```yaml +name: my-workflow +max_parallel_executions: 5 +on_limit: queue # "queue" (default) | "reject" + +steps: + - name: process-batch + action: http/request + depends_on: [split-data] + max_parallel: 3 + params: ... +``` + +`mantle.yaml`: +```yaml +engine: + max_concurrent_executions_per_team: 50 +``` + +### New Execution Status + +``` +queued → pending → running → {completed | failed | cancelled} +``` + +- `queued` = waiting for a concurrency slot +- `cancelled` works from both `queued` and `running` states +- `queued → cancelled` must set `completed_at` (same as other terminal transitions) + +**Code paths that need `queued` awareness:** +- `createExecution` must support inserting as `queued` (currently always inserts as `pending`) +- `updateExecutionStatus` terminal check (engine.go) must recognize `queued` as a valid non-terminal state +- `mantle cancel` must handle cancelling `queued` executions (no steps to cancel, just status update) +- `mantle status` must display `queued` status + +### Enforcement — Advisory Locks + +Execution creation wraps in a transaction: + +1. `pg_advisory_xact_lock(hash(team_id))` → count running executions for team → check per-team limit +2. `pg_advisory_xact_lock(hash(workflow_name))` → count running executions for workflow → check per-workflow limit +3. Under limits → insert with `status = 'pending'` +4. Over limits, `on_limit = queue` → insert with `status = 'queued'` +5. Over limits, `on_limit = reject` → return error (webhook returns 429) + +### Queue Promotion + +- **Primary (completion-triggered):** When an execution reaches a terminal state, `promoteQueued()` checks for the oldest `queued` execution for the same workflow and promotes it to `pending`. +- **Backup (30s poller):** Scans for queued executions with available slots. Catches crashes during promotion. + +### Fan-Out max_parallel in ReadySteps + +`ReadySteps` (orchestrator.go) currently returns all steps whose deps are resolved. For steps that share a fan-out parent, count how many are currently `running` and only release up to `max_parallel`. Requires passing `MaxParallel` into the `workflowStep` struct. + +### Webhook Behavior + +- `on_limit: queue` (default) → 202 Accepted, execution created as `queued` +- `on_limit: reject` → 429 Too Many Requests, no execution created + +### CLI Behavior + +- `mantle run` respects concurrency limits by default +- `mantle run --force` bypasses limits (operator override) + +### Migration + +- Add `max_concurrent_executions INT` nullable column to `teams` table (overrides config default) +- No changes to `workflow_definitions` — concurrency fields live in the YAML content stored as JSONB + +### Struct Changes + +- `workflow.Workflow`: Add `MaxParallelExecutions int`, `OnLimit string`. `OnLimit` zero value (`""`) is treated as `"queue"` during enforcement — the default is applied in the enforcement logic, not during parsing. +- `workflow.Step`: Add `MaxParallel int` +- `config.EngineConfig`: Add `MaxConcurrentExecutionsPerTeam int` + +### Metrics + +- `mantle_executions_queued` (gauge) — current queue depth per workflow +- `mantle_executions_rejected_total` (counter) — rejected due to limit +- `mantle_queue_wait_duration_seconds` (histogram) — time in queue before promotion + +### Edge Cases + +- Any terminal state frees a concurrency slot +- Orphaned `running` executions after crash count against limits temporarily — accepted tradeoff; reaper handles recovery, backup poller covers queue promotion +- `0` means unlimited for all three dimensions + +--- + +## 5. on_failure Workflow Lifecycle Hooks (#30) + +Three lifecycle hook blocks that execute after main workflow steps complete. + +### Hook Blocks + +| Hook | Fires when | Use case | +|------|-----------|----------| +| `on_success` | Main workflow completes with no unhandled failures | Success notifications, downstream triggers | +| `on_failure` | Main workflow step fails (without `continue_on_error`) or workflow times out | Alerts, cleanup, incident creation | +| `on_finish` | Always — success, failure, or timeout (but NOT cancellation) | Resource cleanup, audit logging | + +### YAML Schema + +```yaml +name: my-workflow +timeout: 5m +steps: + - name: fetch-data + action: http/request + params: + url: https://api.example.com/data + +hooks: + timeout: 2m + on_success: + - name: notify-team + action: slack/send + credential: slack-bot + params: + channel: "#ops" + text: "Workflow completed successfully" + on_failure: + - name: alert-ops + action: slack/send + credential: slack-bot + params: + channel: "#ops-alerts" + text: "Failed at {{ execution.failed_step }}: {{ execution.error }}" + on_finish: + - name: cleanup + action: http/request + params: + method: POST + url: "https://api.example.com/cleanup" +``` + +### Execution Order (Fixed) + +``` +Main workflow succeeds → on_success → on_finish +Main workflow fails → on_failure → on_finish +Main workflow timeout → on_failure → on_finish +Main workflow cancel → no hooks +``` + +1. Conditional hook runs first (`on_success` OR `on_failure`, never both from main workflow) +2. `on_finish` always runs last (except on cancellation) + +### Struct Changes + +Add `Hooks *HooksConfig` to `workflow.Workflow`: + +```go +type HooksConfig struct { + Timeout string `yaml:"timeout"` + OnSuccess []Step `yaml:"on_success"` + OnFailure []Step `yaml:"on_failure"` + OnFinish []Step `yaml:"on_finish"` +} +``` + +Hook steps reuse the existing `Step` type with full feature support: `credential`, `timeout`, `retry`, `if`, `continue_on_error`, `artifacts`, `registry_credential`. + +### Hook Step Behavior + +- Flat list, executed sequentially — no `depends_on` +- Names unique within their block, not globally (can share names with main steps). The `hook_block` column provides disambiguation in `step_executions` — queries must always filter on `hook_block IS NULL` for main steps or `hook_block = $2` for hook steps. +- Hook step fails → halt remaining steps in that block → next block still runs +- `continue_on_error: true` on a hook step → record failure, continue to next step in block +- `on_finish` always runs regardless of conditional block outcome +- No cascading: `on_success` failure does NOT trigger `on_failure` +- Hook failures are logged, audited, and emitted as metrics +- Hook failures never alter the workflow execution status + +### CEL Context in Hooks + +| Variable | Description | +|----------|-------------| +| `steps['name'].output` | Main workflow step outputs | +| `steps['name'].error` | Main workflow step errors | +| `hooks['name'].output` | Hook step outputs (accumulates across blocks) | +| `hooks['name'].error` | Hook step errors | +| `execution.status` | `"completed"`, `"failed"`, or `"timed_out"` | +| `execution.error` | Error string or `null` | +| `execution.failed_step` | Name of the step that caused failure, or `null` | +| `execution.failed_in` | `"steps"`, `"on_success"`, `"on_failure"`, `"on_finish"`, or `null` | +| `inputs`, `trigger`, `env`, `artifacts` | Same as main workflow steps | + +### Engine Changes + +After `resumeExecution` completes the main steps, a new `executeHooks` method runs: + +1. Determine main workflow outcome: `completed`, `failed`, or `timed_out` +2. Build execution context CEL variables (`execution.status`, etc.) +3. Run the conditional block (`on_success` or `on_failure`) +4. Run `on_finish` +5. Each block respects `hooks.timeout` as an aggregate cap + +### Timeouts + +- `timeout` (workflow-level, existing) — covers main steps only +- `hooks.timeout` (optional) — separate aggregate cap for all hook execution. The hook timeout starts fresh when hooks begin executing, regardless of how much time the main workflow consumed. If the main workflow times out, hooks get their full `hooks.timeout` budget. +- Individual hook steps can have their own `timeout` (existing field) + +### DB Representation + +Hook step executions go into `step_executions` with a `hook_block TEXT` column: +- `NULL` for main steps +- `"on_success"`, `"on_failure"`, or `"on_finish"` for hook steps + +This keeps existing claim/lease/reaper infrastructure working for hook steps. + +**Important:** `GetStepStatuses` in `orchestrator.go` must add `AND hook_block IS NULL` to its query filter. Without this, hook steps would pollute the DAG-based `AdvanceExecution` logic. Hook step statuses are queried separately by the `executeHooks` method using `AND hook_block = $2`. + +### Validation Additions + +- Hook step names unique within their block +- No `depends_on` in hook steps (reject with clear error) +- `hooks.timeout` is valid duration if present + +### Metrics + +- `mantle_hook_steps_total{workflow, hook, step, status}` counter +- `mantle_hook_steps_failed_total{workflow, hook, step}` counter + +### Concurrency Interaction + +The concurrency slot (from #49) is held from `running` through hook completion. Queue promotion fires when the execution reaches its terminal state, which is after all hooks complete. This prevents a queued execution from starting while the previous execution's cleanup hooks are still running. + +### What This Does NOT Do + +- Hooks do not cascade (`on_success` failure does not trigger `on_failure`) +- Hooks do not alter workflow execution status +- Hooks do not support `depends_on` (use child workflows for complex error handling) +- `continue_on_error` workflows that "complete" with step errors do NOT trigger `on_failure` + +--- + +## 6. Retry from Failed Step (#48) + +New CLI command to resume execution from the point of failure. + +### Command + +``` +mantle retry [--from-step ] [--force] +``` + +- `--from-step` retries from a specific step regardless of its status (failed, cancelled, or completed). If omitted, retries from the first failed step in topological order. +- `--force` bypasses concurrency limits (consistent with `mantle run --force`). + +### Flow + +1. Load original execution from `workflow_executions` (workflow name, version, inputs) +2. Load all step statuses via `GetStepStatuses` +3. If `--from-step` provided, validate step exists; otherwise find the first failed step in topological order +4. Create a **new execution** (new UUID, same workflow version, same inputs) +5. Copy completed step outputs from the original execution for all steps preceding the retry point — insert as `completed` step_executions in the new execution. **Do not copy hook steps** (`hook_block IS NOT NULL`); hooks fire fresh based on the new outcome. +6. Mark the retry-point step and all downstream steps as `pending` +7. Execute the new workflow via `Engine.Execute()` (respects concurrency limits unless `--force`) +8. Emit `ActionExecutionRetried` audit event with execution ID, target step, and operator + +### Design Decisions + +- **New execution, not resume:** The old execution is a completed audit record. Creating a new execution with `retried_from` metadata preserves the audit trail. +- **Same version:** Retry uses the workflow version from the original execution, not the latest version. Users who want the new definition should `mantle run` fresh. +- **Hooks fire fresh:** Since this is a new execution, hooks run based on the new outcome, not the original. + +### DB Changes + +- Add `retried_from_execution_id UUID` nullable column to `workflow_executions` + +### CLI Output + +Same as `mantle run` — streams logs, returns final status. + +### Changes + +- `internal/cli/retry.go`: New command +- `internal/engine/engine.go`: Add `RetryExecution(ctx, originalExecID, fromStep)` method +- `internal/audit/actions.go`: Add `ActionExecutionRetried` + +--- + +## 7. Workflow Rollback (#50) + +New CLI command to revert a workflow to a previous version. + +### Command + +``` +mantle rollback [--to-version ] +``` + +### Flow + +1. If `--to-version` provided, use it; otherwise default to the **second most recent version** (not literal `current - 1`, since version numbers may be non-contiguous due to prior rollbacks) +2. Validate target version exists in `workflow_definitions` +3. Reject version `0` or negative values +4. Load target version's content +5. Validate target version content differs from current (reject no-op rollback with message) +6. Insert a **new version row**: `version = current_max + 1`, content = target version's content, `rollback_of = target_version` +7. If running in server mode, reload triggers for the updated workflow +8. Emit `ActionWorkflowRolledBack` audit event with from/to version info +9. Print: `"Rolled back from version N to content of version M (now version N+1)"` + +### Design Decisions + +- **New version, not pointer:** The engine always runs the latest version. Creating a new row with old content preserves full version history and audit trail. +- **`mantle plan` shows no diff** after rollback (content matches what's deployed). +- **In-flight executions unaffected:** Running executions use their original version. The rollback only affects new executions. Rollback is **not blocked** by running executions. +- **Trigger reload:** Server mode must pick up trigger changes (cron schedules, webhook paths) from the rolled-back content. +- **Concurrency interaction:** Rollback does not interact with concurrency controls — it only inserts a new version row. + +### DB Changes + +- Add `rollback_of INT` nullable column to `workflow_definitions` (points to the version number whose content was restored, for traceability) + +### Changes + +- `internal/cli/rollback.go`: New command +- `internal/audit/actions.go`: Add `ActionWorkflowRolledBack` + +--- + +## Migration Summary + +Single migration file `016_safety_net.sql` covering all DB changes: + +```sql +-- Concurrency controls (#49) +ALTER TABLE teams ADD COLUMN max_concurrent_executions INT; + +-- Retry from failed step (#48) +ALTER TABLE workflow_executions ADD COLUMN retried_from_execution_id UUID + REFERENCES workflow_executions(id); + +-- Lifecycle hooks (#30) +ALTER TABLE step_executions ADD COLUMN hook_block TEXT; + +-- Workflow rollback (#50) +ALTER TABLE workflow_definitions ADD COLUMN rollback_of INT; +``` + +--- + +## Implementation Order Rationale + +1. **#52 Config versioning** + **#75 Rename tmp→storage** — Low-risk config changes. #52 provides the versioning guard rail; #75 ships the breaking rename with deprecation fallback. Both touch `config.go`. +2. **#51 Secret key rotation** — Standalone security feature. No dependencies on other issues. +3. **#49 Concurrency controls** — Adds `queued` status and changes the execution creation path. Must land before hooks so hooks understand the full status lifecycle. +4. **#30 on_failure hooks** — Largest feature. Needs to know about all execution statuses including `queued` from #49. +5. **#48 Retry** + **#50 Rollback** — Additive CLI commands. Benefit from the updated execution model but don't change the core engine path. diff --git a/docs/superpowers/specs/2026-03-27-env-map-config-design.md b/docs/superpowers/specs/2026-03-27-env-map-config-design.md new file mode 100644 index 0000000..66a7ddd --- /dev/null +++ b/docs/superpowers/specs/2026-03-27-env-map-config-design.md @@ -0,0 +1,86 @@ +# env: Map in mantle.yaml — Design Spec + +**Issue:** #74 +**Milestone:** v0.5.0 — The GitOps Update + +--- + +## Summary + +Add an optional `env:` section to `mantle.yaml` that populates the `env.*` namespace in CEL expressions. Merges with `MANTLE_ENV_*` shell environment variables at runtime, with env vars taking precedence. Logs when an env var overrides a YAML value. + +## Schema + +```yaml +version: 1 +env: + SLACK_CHANNEL: "#ops-alerts" + ENVIRONMENT: production + NOTIFY_EMAIL: "team@example.com" +database: + url: postgres://... +``` + +Values are accessible in CEL as `env.SLACK_CHANNEL`, `env.ENVIRONMENT`, etc. — same namespace as existing `MANTLE_ENV_*` variables. + +## Merge Behavior + +1. Start with `cfg.Env` map from mantle.yaml +2. Overlay `MANTLE_ENV_*` from `os.Environ()` (strip prefix as today) +3. When a key exists in both, the env var wins and an `slog.Info` is emitted: `"env variable MANTLE_ENV_ overrides config env."` + +## Changes + +### `internal/config/config.go` + +Add field to `Config` struct: + +```go +Env map[string]string `mapstructure:"env"` +``` + +No Viper env binding — the `env:` section is a free-form map read from the unmarshaled struct only. No `v.SetDefault` needed (nil map is fine — merge handles it). + +### `internal/cel/cel.go` + +Change `envVars()` to accept the config env map: + +```go +func envVars(configEnv map[string]string) map[string]string +``` + +1. Copy `configEnv` into result map (nil-safe) +2. Iterate `os.Environ()` for `MANTLE_ENV_*` prefix +3. For each match, check if key already exists in result — if so, `slog.Info` override message +4. Set the value (env var wins) + +Update the `NewEvaluator` constructor (or add a method) to accept the config env map and pass it to `envVars()`. The evaluator caches env vars at construction time, so the config map is needed at that point. + +### Engine wiring + +Where the CEL evaluator is created (in `engine.New()` or the CLI commands that construct engines), pass `cfg.Env` to the evaluator. This may require: +- Adding a `ConfigEnv` parameter to `NewEvaluator`, or +- Adding a `WithConfigEnv(map[string]string)` option, or +- Passing it through the `Engine` struct + +Given that `engine.New()` currently takes only `*sql.DB`, the simplest approach is to update `NewEvaluator` to accept the config env map: `NewEvaluator(configEnv map[string]string)`. Then pass `cfg.Env` at each call site where the evaluator is created (CLI commands and serve). This avoids changing the `Engine.New()` signature. + +### Documentation + +Update `packages/site/src/content/docs/configuration.md`: +- Document the `env:` section +- Explain merge precedence (env vars > YAML) +- Example showing override behavior + +### Tests + +- `cel_test.go`: Test that config env values appear in CEL context +- `cel_test.go`: Test that `MANTLE_ENV_*` overrides config env +- `config_test.go`: Test that `env:` map parses from YAML + +## What This Does NOT Do + +- No per-workflow env scoping (engine-wide only) +- No env var binding for individual `env.*` keys via Viper +- No secret support in `env:` values (use `credential:` for secrets) +- No validation of env key names (free-form strings) diff --git a/docs/superpowers/specs/2026-03-27-workflow-composition-design.md b/docs/superpowers/specs/2026-03-27-workflow-composition-design.md new file mode 100644 index 0000000..33d2610 --- /dev/null +++ b/docs/superpowers/specs/2026-03-27-workflow-composition-design.md @@ -0,0 +1,182 @@ +# Workflow Composition — workflow/run Action: Design Spec + +**Issue:** #54 +**Milestone:** v0.5.0 — The GitOps Update + +--- + +## Summary + +Add a `workflow/run` connector that lets one workflow invoke another synchronously. Child execution blocks until complete, output is wrapped and accessible to the parent, cancellation cascades, and token budgets are inherited or capped. + +## YAML Schema + +```yaml +- name: send-notification + action: workflow/run + params: + workflow: notification-sender + version: 3 # optional, defaults to latest + inputs: + channel: "{{ steps['classify'].output.json.category }}" + message: "{{ steps['classify'].output.json.summary }}" + token_budget: 5000 # optional, defaults to parent's remaining budget +``` + +## Output + +The step output wraps the child's full execution result: + +```json +{ + "execution_id": "abc-123", + "status": "completed", + "steps": { + "step-1": { "output": { ... } }, + "step-2": { "output": { ... } } + } +} +``` + +Parent accesses child results: `steps['send-notification'].output.steps['send-email'].output.message_id` + +## Migration (016 on main, 017 if v0.4.0 lands first) + +```sql +ALTER TABLE workflow_executions ADD COLUMN parent_execution_id UUID REFERENCES workflow_executions(id); +ALTER TABLE workflow_executions ADD COLUMN parent_step_name TEXT; +ALTER TABLE workflow_executions ADD COLUMN depth INTEGER NOT NULL DEFAULT 0; +CREATE INDEX idx_workflow_executions_parent ON workflow_executions(parent_execution_id) WHERE parent_execution_id IS NOT NULL; +``` + +## Engine Config + +```yaml +engine: + max_workflow_depth: 10 # default +``` + +Ensure `EngineConfig.MaxWorkflowDepth` uses default 10 and is bound to env/config: + +```go +v.SetDefault("engine.max_workflow_depth", 10) +_ = v.BindEnv("engine.max_workflow_depth", "MANTLE_ENGINE_MAX_WORKFLOW_DEPTH") +``` + +## Connector Implementation + +New file: `internal/connector/workflow.go` + +The `workflow/run` connector is special — it needs access to the `Engine` to execute the child workflow. Unlike other connectors which are stateless functions, this one needs a reference back to the engine. + +**Approach:** The `workflow/run` connector cannot be registered inside `NewRegistry()` because it needs an `*Engine` reference, which doesn't exist yet at registry construction time. Instead, register it after the engine is constructed — in the CLI commands (run.go, serve.go) or in a new `Engine.RegisterBuiltinConnectors()` method. The connector struct holds an engine pointer: + +```go +type WorkflowConnector struct { + engine *Engine // set after engine construction +} +``` + +This avoids circular imports between `engine` and `connector` packages. + +**Execute logic:** +1. Resolve `params.workflow` name and optional `params.version` (latest if omitted) +2. Check depth: query parent's depth from DB, reject if `depth + 1 > max_workflow_depth` +3. Resolve inputs via CEL (already handled by the engine's param resolution) +4. Call `engine.Execute()` (or a variant) with: + - The child workflow name and version + - The resolved inputs + - Parent's team context (same team_id) + - `parent_execution_id` and `parent_step_name` set on the new execution + - Token budget: `params.token_budget` if set, else parent's remaining budget +5. Return the wrapped `ExecutionResult` as the step output + +## Recursion Safety + +- `max_workflow_depth` checked at child creation time +- Depth stored on `workflow_executions.depth` column +- Self-recursion is allowed (depth limit is the only guard) +- Error message: `"workflow depth limit exceeded (%d/%d)"` with current depth and max + +## Cancellation Cascade + +When `mantle cancel ` is called: +- Cancel the parent execution (existing behavior) +- Cancel the entire execution tree atomically using a recursive CTE: + +```sql +WITH RECURSIVE children AS ( + SELECT id FROM workflow_executions WHERE id = $1 + UNION ALL + SELECT e.id FROM workflow_executions e + JOIN children c ON e.parent_execution_id = c.id +) +UPDATE workflow_executions SET status = 'cancelled', completed_at = NOW(), updated_at = NOW() +WHERE id IN (SELECT id FROM children) AND status IN ('pending', 'running', 'queued') +``` + +This handles arbitrarily deep trees atomically, avoiding TOCTOU races where new children could be created between loop iterations. + +Update `internal/cli/cancel.go` to use this CTE. + +## Checkpoint Recovery + +On crash recovery (parent resumes from checkpoint): +- Before creating a new child execution, check if one already exists for this `parent_execution_id + parent_step_name` +- If found and still running/pending, resume it (wait for completion) +- If found and completed, use its result (skip re-execution) +- If found and failed, treat as step failure (don't re-create) + +This is handled in the connector's Execute logic by querying for existing child executions first. + +## Token Budget + +- Child inherits parent's context (including `StepContext.WorkflowTokenBudget`) +- If `params.token_budget` is set, use `min(params.token_budget, parent_remaining)` as the child's budget +- Child's AI token usage counts against whatever budget it was given + +## Credential Scoping + +- Child runs in parent's team context (same `team_id` from context) +- Credentials resolve against the calling team's store +- No cross-team credential access (enforced by existing team_id scoping) + +## CLI Enhancements + +### `mantle logs ` +- When displaying steps, check if a step used `workflow/run` +- If so, query child execution by `parent_execution_id + parent_step_name` +- Display child steps indented under the parent step +- `--shallow` flag suppresses child details + +### `mantle status ` +- Show execution tree: parent → children with status + +## Feature Interactions + +- **continue_on_error:** Child failure = step failure. Parent continues if `continue_on_error: true`. +- **Retry policies:** Parent step retry re-invokes child from scratch (new child execution). +- **Concurrency controls:** Child execution counts against per-workflow limits for the *child* workflow name. +- **Lifecycle hooks:** Child hooks run within the child execution. Parent hooks run after parent completes. +- **Artifacts:** Child artifacts are accessible in the wrapped output. + +## Changes Summary + +| Action | File | Responsibility | +|--------|------|----------------| +| Create | `internal/db/migrations/017_workflow_composition.sql` | Schema changes | +| Create | `internal/connector/workflow.go` | workflow/run connector | +| Modify | `internal/connector/connector.go` | Register workflow/run | +| Modify | `internal/config/config.go` | Add MaxWorkflowDepth | +| Modify | `internal/engine/engine.go` | ExecuteChild method, depth tracking | +| Modify | `internal/cli/cancel.go` | Cascade cancellation to children | +| Modify | `internal/cli/logs.go` | Show child logs inline | +| Modify | `internal/cli/status.go` | Show execution tree | +| Modify | `internal/audit/audit.go` | Add ActionChildWorkflowExecuted | +| Update | Docs: workflow-reference, connectors | Document workflow/run | + +## What This Does NOT Do + +- Async invocation (fire-and-forget, fan-out/fan-in of child workflows) +- Cross-team workflow calls +- Input validation at invocation time (child validates its own inputs) diff --git a/packages/engine/.goreleaser.yaml b/packages/engine/.goreleaser.yaml index acd5dfa..5a7eec0 100644 --- a/packages/engine/.goreleaser.yaml +++ b/packages/engine/.goreleaser.yaml @@ -22,8 +22,14 @@ builds: archives: - id: default - format: tar.gz + formats: ["tar.gz"] name_template: "mantle-{{ .Os }}-{{ .Arch }}" + files: + # Bundle the license so every distributed archive satisfies BUSL-1.1's + # notice requirement. Path is relative to the goreleaser workdir + # (packages/engine); LICENSE lives at the repo root. + - src: "../../LICENSE" + dst: "LICENSE" checksum: name_template: checksums.txt @@ -34,51 +40,26 @@ changelog: groups: - title: Features regexp: '^.*?feat(\(.+\))?\!?:.*$' + order: 0 - title: Bug Fixes regexp: '^.*?fix(\(.+\))?\!?:.*$' + order: 1 - title: Other order: 999 -dockers: - - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-amd64" - use: buildx - ids: [mantle] - goos: linux - goarch: amd64 +dockers_v2: + - ids: [mantle] dockerfile: Dockerfile.goreleaser - build_flag_templates: - - "--platform=linux/amd64" - - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-arm64" - use: buildx - ids: [mantle] - goos: linux - goarch: arm64 - dockerfile: Dockerfile.goreleaser - build_flag_templates: - - "--platform=linux/arm64" - -docker_manifests: - - name_template: "ghcr.io/dvflw/mantle:{{ .Version }}" - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-amd64" - - "ghcr.io/dvflw/mantle:{{ .Version }}-arm64" - - name_template: "ghcr.io/dvflw/mantle:{{ .Major }}.{{ .Minor }}" - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-amd64" - - "ghcr.io/dvflw/mantle:{{ .Version }}-arm64" - skip_push: auto - - name_template: "ghcr.io/dvflw/mantle:{{ .Major }}" - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-amd64" - - "ghcr.io/dvflw/mantle:{{ .Version }}-arm64" - skip_push: auto - - name_template: "ghcr.io/dvflw/mantle:latest" - image_templates: - - "ghcr.io/dvflw/mantle:{{ .Version }}-amd64" - - "ghcr.io/dvflw/mantle:{{ .Version }}-arm64" - skip_push: auto + images: + - "ghcr.io/dvflw/mantle" + tags: + - "{{ .Version }}" + # Floating tags (major.minor, major, latest) are only pushed for stable + # releases. Empty template results are skipped by goreleaser, so these + # evaluate to "" and are dropped for any pre-release version. + - "{{ if not .Prerelease }}{{ .Major }}.{{ .Minor }}{{ end }}" + - "{{ if not .Prerelease }}{{ .Major }}{{ end }}" + - "{{ if not .Prerelease }}latest{{ end }}" release: github: diff --git a/packages/engine/Dockerfile.goreleaser b/packages/engine/Dockerfile.goreleaser index fe58106..371412d 100644 --- a/packages/engine/Dockerfile.goreleaser +++ b/packages/engine/Dockerfile.goreleaser @@ -4,7 +4,8 @@ RUN apk add --no-cache ca-certificates tzdata \ && addgroup -S mantle \ && adduser -S -G mantle -h /home/mantle -s /bin/sh mantle -COPY mantle /usr/local/bin/mantle +ARG TARGETPLATFORM +COPY $TARGETPLATFORM/mantle /usr/local/bin/mantle ENV MANTLE_LOG_LEVEL=info ENV MANTLE_API_ADDRESS=:8080