diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 05787db..d1f766b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,9 +2,8 @@ name: CI on: push: - branches: ['**'] + branches: [main] pull_request: - branches: ['**'] jobs: test: diff --git a/README.md b/README.md index a418a39..ad00c59 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ steps: - **Checkpoint and Resume** -- Steps checkpoint to Postgres after completion. Crash mid-workflow? Restart picks up from the last completed step. - **Secrets as Opaque Handles** -- Credentials are AES-256-GCM encrypted at rest and resolved at connector invocation time. Never exposed in expressions, logs, or step outputs. -[How Mantle compares to Temporal, n8n, LangChain, Airflow, and others.](/docs/comparison) +[How Mantle compares to Temporal, n8n, LangChain, Airflow, and others.](https://mantle.dvflw.co/docs/comparison) ## Quick Start @@ -69,7 +69,7 @@ mantle apply examples/hello-world.yaml mantle run hello-world ``` -17 example workflows are included in [`examples/`](examples/). See the [Getting Started guide](/docs/getting-started) for a full walkthrough. +17 example workflows are included in [`examples/`](examples/). See the [Getting Started guide](https://mantle.dvflw.co/docs/getting-started) for a full walkthrough. ## Connectors @@ -85,7 +85,7 @@ mantle run hello-world | Postgres | `postgres/query` | Parameterized SQL against external databases | | S3 | `s3/put`, `s3/get`, `s3/list` | Put, get, list objects (S3-compatible) | -Need something else? Write a [plugin](/docs/plugins). Any executable that reads JSON from stdin and writes JSON to stdout -- Python, Rust, Node, Bash. +Need something else? Write a [plugin](https://mantle.dvflw.co/docs/plugins-guide). Any executable that reads JSON from stdin and writes JSON to stdout -- Python, Rust, Node, Bash. ## CLI Reference @@ -132,7 +132,7 @@ Need something else? Write a [plugin](/docs/plugins). Any executable that reads | `mantle cleanup` | Clean up old executions and artifacts | | `mantle version` | Print version info | -See the [CLI Reference](/docs/cli-reference) for full usage and flags. +See the [CLI Reference](https://mantle.dvflw.co/docs/cli-reference) for full usage and flags. ## Configuration @@ -204,7 +204,7 @@ make clean # Remove built binary ## Contributing -Contributions are welcome. Please open an issue first to discuss what you want to change. See the [Contributing guide](/docs/contributing) for details. +Contributions are welcome. Please open an issue first to discuss what you want to change. See [CONTRIBUTING.md](CONTRIBUTING.md) for details. ## License diff --git a/docs/authentication-guide.md b/docs/authentication-guide.md deleted file mode 100644 index 6f02574..0000000 --- a/docs/authentication-guide.md +++ /dev/null @@ -1,338 +0,0 @@ -# Authentication & RBAC Guide - -Mantle supports two authentication methods: API keys and OIDC/SSO (JWT). Both resolve to the same User model with team scoping. This guide covers setting up authentication, configuring roles, and managing teams. - -For CLI flag details, see the [CLI Reference](cli-reference.md). For the server endpoints that require authentication, see the [Server Guide](server-guide.md). - -## Authentication Methods at a Glance - -| Method | Format | Best For | -|---|---|---| -| API Key | `mk_` + 64 hex chars | CI pipelines, scripts, programmatic access | -| OIDC/SSO | JWT from your identity provider | Interactive users, browser-based access, SSO enforcement | - -Both methods produce the same result: a resolved user with a team, a role, and an audit trail. Every state-changing API request is logged with the authenticated user's identity. - -## API Key Authentication - -API keys are the simplest way to authenticate with Mantle. Each key is tied to a specific user and inherits that user's role and team membership. - -### How Keys Work - -When you create an API key, Mantle generates a random key with the `mk_` prefix and displays it once. The raw key is never stored -- Mantle keeps only a SHA-256 hash in the database. On each request, the server hashes the provided key and looks up the matching record. - -### Creating an API Key - -Create a key for an existing user: - -```bash -mantle users api-key --email alice@example.com --key-name prod-key -``` - -``` -API Key: mk_a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2 - -Save this key — it cannot be retrieved again. -Key prefix for reference: mk_a1b2c3 -``` - -The `--key-name` flag is a human-readable label for identifying the key later (e.g., in audit logs or when revoking). - -### Using an API Key - -Pass the key in one of three ways, listed by precedence: - -**HTTP header (recommended for API calls):** - -```bash -curl -H "Authorization: Bearer mk_a1b2c3..." http://localhost:8080/api/v1/executions -``` - -**CLI flag:** - -```bash -mantle run my-workflow --api-key mk_a1b2c3... -``` - -**Environment variable:** - -```bash -export MANTLE_API_KEY="mk_a1b2c3..." -mantle run my-workflow -``` - -The CLI flag overrides the environment variable. The HTTP header is the only option for REST API calls. - -### Key Lifecycle - -**Expiration.** Keys have an optional `expires_at` timestamp. Create a key with an expiration: - -```bash -mantle users api-key --email alice@example.com --key-name temp-key --expires 90d -``` - -Expired keys are rejected at authentication time. The key remains in the database for audit purposes. - -**Last-used tracking.** Mantle updates the `last_used_at` timestamp on every successful authentication. Use this to identify stale keys: - -```bash -mantle users list-keys --email alice@example.com -``` - -``` -NAME PREFIX CREATED LAST USED EXPIRES -prod-key mk_a1b2c3 2026-03-18 14:30:00 2026-03-22 09:15:00 never -temp-key mk_d4e5f6 2026-03-20 10:00:00 never 2026-06-18 10:00:00 -``` - -**Revocation.** Revoke a key immediately: - -```bash -mantle auth revoke-key --email alice@example.com --key-name prod-key -``` - -``` -Revoked key "prod-key" for alice@example.com -``` - -Revoked keys are rejected on the next request. There is no grace period. - -### Key Format - -All Mantle API keys follow the format `mk_` followed by 64 hexadecimal characters. The `mk_` prefix makes keys easy to identify in logs and secret scanners. - -## OIDC/SSO Authentication - -Mantle supports OpenID Connect for single sign-on with identity providers like Okta, Auth0, Azure AD, and Google Workspace. - -### Configuration - -Configure OIDC in `mantle.yaml`: - -```yaml -auth: - oidc: - issuer_url: "https://company.okta.com/oauth2/default" - client_id: "0oa1234..." - client_secret: "" # optional for public clients - audience: "mantle" - allowed_domains: ["company.com"] -``` - -Or use environment variables: - -```bash -export MANTLE_AUTH_OIDC_ISSUER_URL="https://company.okta.com/oauth2/default" -export MANTLE_AUTH_OIDC_CLIENT_ID="0oa1234..." -export MANTLE_AUTH_OIDC_CLIENT_SECRET="" -export MANTLE_AUTH_OIDC_AUDIENCE="mantle" -export MANTLE_AUTH_OIDC_ALLOWED_DOMAINS="company.com" -``` - -For multiple allowed domains, separate them with commas in the environment variable: `MANTLE_AUTH_OIDC_ALLOWED_DOMAINS="company.com,subsidiary.com"`. - -### How It Works - -1. The user authenticates with your identity provider and obtains a JWT -2. The JWT is sent to Mantle in the `Authorization: Bearer ` header -3. Mantle validates the JWT signature against the provider's JWKS endpoint (fetched from `issuer_url/.well-known/openid-configuration`) -4. Mantle verifies the `aud` (audience), `iss` (issuer), and `exp` (expiration) claims -5. The `email` claim is extracted and matched against a pre-provisioned user in the database -6. The `email_verified` claim must be `true` -- unverified emails are rejected -7. The email domain is checked against `allowed_domains` if configured - -### User Pre-Provisioning - -Users must be created in Mantle before they can authenticate via OIDC. There is no just-in-time (JIT) provisioning -- this is intentional to prevent unauthorized access from anyone with a valid identity provider account. - -Create users before they log in: - -```bash -mantle users create --email alice@company.com --name "Alice Chen" --team engineering --role operator -``` - -If a valid JWT arrives for an email address that does not exist in Mantle, the request is rejected with a 403 Forbidden response. - -### CLI Login Flow - -For interactive CLI use, Mantle supports two OIDC login flows: - -**Browser-based (authorization code with PKCE):** - -```bash -mantle login -``` - -This opens your default browser to the identity provider's login page. After authentication, the browser redirects to a local callback server and Mantle stores the token. - -**Device flow (headless environments):** - -```bash -mantle login --device -``` - -This prints a URL and a code. Open the URL on any device, enter the code, and authenticate. Useful for SSH sessions and CI environments. - -**Logout:** - -```bash -mantle logout -``` - -This removes the cached credentials. - -### Credential Caching - -After a successful `mantle login`, the JWT and refresh token are stored at `~/.mantle/credentials`. The CLI automatically refreshes expired tokens using the refresh token. If the refresh token is also expired, you are prompted to log in again. - -The credentials file is created with `0600` permissions (owner read/write only). - -## RBAC - -Mantle uses role-based access control with three roles. Roles are hierarchical -- each role includes all permissions of the roles below it. - -### Roles and Permissions - -| Permission | `operator` | `team_owner` | `admin` | -|---|---|---|---| -| Run workflows | Yes | Yes | Yes | -| View executions and logs | Yes | Yes | Yes | -| Cancel executions | Yes | Yes | Yes | -| Apply workflow definitions | No | Yes | Yes | -| Create and delete credentials | No | Yes | Yes | -| Manage team members | No | Yes | Yes | -| Assign roles within team | No | Yes (operator only) | Yes | -| Create and delete teams | No | No | Yes | -| Manage all users | No | No | Yes | -| Revoke API keys for any user | No | No | Yes | -| View audit logs | No | No | Yes | -| Access `/metrics` endpoint | No | No | Yes | - -**`operator`** -- the default role. Can run workflows, view results, and cancel executions. Cannot modify workflow definitions, credentials, or team membership. - -**`team_owner`** -- manages a team's workflows and credentials. Can apply workflow definitions, create credentials, and add or remove team members. Can assign the `operator` role to team members. - -**`admin`** -- full access. Can create and delete teams, manage all users across all teams, view audit logs, and access Prometheus metrics. - -### Assigning Roles - -Assign a role to an existing user: - -```bash -mantle roles assign --email alice@example.com --role team_owner -``` - -``` -Assigned role "team_owner" to alice@example.com -``` - -Only admins can assign any role. Team owners can assign the `operator` role to members of their own team. - -### Viewing a User's Role - -```bash -mantle users get --email alice@example.com -``` - -``` -Email: alice@example.com -Name: Alice Chen -Team: engineering -Role: team_owner -Created: 2026-03-18 14:30:00 -``` - -## Teams - -Teams provide data isolation in multi-tenant environments. All workflows, executions, credentials, and audit events are scoped to a team. Users in one team cannot see or interact with another team's data. - -### Creating a Team - -```bash -mantle teams create --name engineering -``` - -``` -Created team engineering (id: f6a7b8c9-d0e1-2345-fabc-456789012345) -``` - -### Listing Teams - -```bash -mantle teams list -``` - -``` -NAME MEMBERS CREATED -engineering 3 2026-03-18 14:30:00 -data-science 2 2026-03-19 10:00:00 -``` - -### Adding Users to a Team - -Specify the team when creating a user: - -```bash -mantle users create --email bob@example.com --name "Bob Park" --team engineering --role operator -``` - -### Data Isolation - -Every database query is scoped by `team_id`. This means: - -- Workflows applied by team A are not visible to team B -- Executions, logs, and step outputs are team-scoped -- Credentials created by team A cannot be referenced by team B's workflows -- Webhook paths are globally unique but only trigger workflows owned by the team that created them - -### Default Team (Single-Tenant Mode) - -When Mantle starts without any teams configured, it operates in single-tenant mode. A `default` team is created automatically during `mantle init`. All users and data belong to this team unless you explicitly create additional teams. - -Single-tenant mode is the starting point described in the [Getting Started](getting-started.md) guide. You can migrate to multi-tenant mode at any time by creating teams and reassigning users. - -## REST API Authentication - -All REST API endpoints require a Bearer token in the `Authorization` header. The token can be either an API key or an OIDC JWT: - -```bash -# With an API key -curl -H "Authorization: Bearer mk_a1b2c3..." http://localhost:8080/api/v1/executions - -# With an OIDC JWT -curl -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIs..." http://localhost:8080/api/v1/executions -``` - -Mantle distinguishes between the two by checking the `mk_` prefix. Tokens starting with `mk_` are treated as API keys; all others are validated as JWTs. - -### Endpoints That Bypass Authentication - -The following endpoints do not require authentication: - -| Endpoint | Purpose | -|---|---| -| `GET /healthz` | Liveness probe | -| `GET /readyz` | Readiness probe | - -### Endpoints That Require Authentication - -All other endpoints require a valid Bearer token: - -| Endpoint | Minimum Role | -|---|---| -| `POST /api/v1/run/{workflow}` | `operator` | -| `GET /api/v1/executions` | `operator` | -| `GET /api/v1/executions/{id}` | `operator` | -| `POST /api/v1/cancel/{id}` | `operator` | -| `POST /hooks/{path}` | `operator` (or unauthenticated if webhook auth is disabled) | -| `GET /metrics` | `admin` | - -Requests with missing, expired, or revoked tokens receive a 401 Unauthorized response. Requests with valid tokens but insufficient role permissions receive a 403 Forbidden response. - -## Further Reading - -- [CLI Reference](cli-reference.md) -- full flag documentation for auth, users, teams, and roles commands -- [Server Guide](server-guide.md) -- production deployment and REST API -- [Configuration](configuration.md) -- all auth-related configuration options and environment variables -- [Secrets Guide](secrets-guide.md) -- credential storage and encryption (separate from authentication) -- [Observability Guide](observability-guide.md) -- audit trail for authentication events diff --git a/docs/cli-reference.md b/docs/cli-reference.md deleted file mode 100644 index 354d66b..0000000 --- a/docs/cli-reference.md +++ /dev/null @@ -1,1200 +0,0 @@ -# CLI Reference - -Mantle is a single binary with subcommands for every stage of the workflow lifecycle. This page documents every command, its flags, and usage examples. - -For installation instructions, see [Getting Started](getting-started.md). - -## Global Flags - -These flags are available on every command (except where noted): - -| Flag | Env Var | Default | Description | -|---|---|---|---| -| `--config` | -- | `mantle.yaml` (in current directory) | Path to the configuration file. | -| `--database-url` | `MANTLE_DATABASE_URL` | `postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable` | Postgres connection URL. | -| `--api-address` | `MANTLE_API_ADDRESS` | `:8080` | API server listen address. | -| `--log-level` | `MANTLE_LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error`. | -| `--api-key` | -- | -- | API key for authentication. Overrides cached credentials from `mantle login`. | -| `--output`, `-o` | -- | `text` | Output format: `text` or `json`. | - -Flag precedence, highest to lowest: CLI flags, environment variables, config file, defaults. See [Configuration](configuration.md) for details. - ---- - -## Working Commands - -### mantle version - -Print version, commit hash, and build date. Does not require a database connection or config file. - -``` -Usage: - mantle version -``` - -**Example:** - -```bash -$ mantle version -mantle v0.1.0 (abc32ad, built 2026-03-18T00:00:00Z) -``` - ---- - -### mantle init - -Run all pending database migrations to set up or upgrade the Mantle schema. This is the first command you run after installing Mantle and starting Postgres. - -``` -Usage: - mantle init -``` - -**Example:** - -```bash -$ mantle init -Running migrations... -Migrations complete. -``` - -**Errors:** - -If Postgres is not running or the connection URL is wrong, you see: - -``` -Error: failed to connect to database: ... -``` - -Fix: verify Postgres is running (`docker-compose up -d`) and check your `--database-url` or `MANTLE_DATABASE_URL`. - ---- - -### mantle migrate - -Run pending database migrations. Functionally equivalent to `mantle init` but organized as a command group with subcommands for status and rollback. - -``` -Usage: - mantle migrate - mantle migrate status - mantle migrate down -``` - -#### mantle migrate status - -Show which migrations have been applied and which are pending. - -```bash -$ mantle migrate status -Applied At Migration -============================================================ -2026-03-18 10:00:00 +0000 001_initial_schema.sql -``` - -#### mantle migrate down - -Roll back the most recently applied migration. Use with caution in production. - -```bash -$ mantle migrate down -Rollback complete. -``` - ---- - -### mantle validate - -Check a workflow YAML file for structural errors. This command is fully offline -- it does not connect to a database or make any network requests. Global config flags are ignored. - -``` -Usage: - mantle validate -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `file` | Yes | Path to the workflow YAML file. | - -**Example (valid workflow):** - -```bash -$ mantle validate workflow.yaml -workflow.yaml: valid -``` - -**Example (invalid workflow):** - -```bash -$ mantle validate bad-workflow.yaml -bad-workflow.yaml:1:1: error: name must match ^[a-z][a-z0-9-]*$ (name) -bad-workflow.yaml: error: at least one step is required (steps) -``` - -Errors include file name, line number, column number, message, and the field path. The command exits with code 1 if any validation errors are found. - -See the [Workflow Reference](workflow-reference.md) for a complete list of validation rules. - ---- - -### mantle apply - -Validate a workflow definition and store it as a new immutable version in the database. If the content has not changed since the last applied version, no new version is created. - -``` -Usage: - mantle apply -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `file` | Yes | Path to the workflow YAML file. | - -**Example (first apply):** - -```bash -$ mantle apply workflow.yaml -Applied fetch-and-summarize version 1 -``` - -**Example (no changes):** - -```bash -$ mantle apply workflow.yaml -No changes — fetch-and-summarize is already at version 1 -``` - -**Example (updated workflow):** - -```bash -$ mantle apply workflow.yaml -Applied fetch-and-summarize version 2 -``` - -The `apply` command: - -1. Reads and parses the YAML file -2. Runs the same validation as `mantle validate` -3. Computes a SHA-256 hash of the file content -4. Compares the hash against the latest stored version -5. If the content changed, inserts a new version row in `workflow_definitions` - -Requires a database connection. If validation fails, nothing is written to the database. - ---- - -### mantle plan - -Diff a local workflow file against the version stored in the database. Shows what will change before you apply. Runs validation first -- if the file is invalid, the diff is not shown. - -``` -Usage: - mantle plan -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `file` | Yes | Path to the workflow YAML file. | - -**Example (new workflow):** - -```bash -$ mantle plan workflow.yaml -+ fetch-and-summarize (new) - -Plan: 1 workflow to create -``` - -**Example (changed workflow):** - -```bash -$ mantle plan workflow.yaml -~ fetch-and-summarize (version 1 → 2) - ~ steps[1].params.model: gpt-4 → gpt-4o - -Plan: 1 workflow to update (version 1 → 2) -``` - -**Example (no changes):** - -```bash -$ mantle plan workflow.yaml -No changes — fetch-and-summarize is at version 2 -``` - -Requires a database connection. - ---- - -### mantle run - -Execute a workflow by name. Uses the latest applied version. - -``` -Usage: - mantle run [flags] -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `workflow` | Yes | Name of the workflow to run. Must have been previously applied. | - -**Flags:** - -| Flag | Description | -|---|---| -| `--input KEY=VALUE` | Pass an input parameter. Can be specified multiple times. | - -**Example:** - -```bash -$ mantle run fetch-and-summarize --input url=https://api.example.com/data -Running fetch-and-summarize (version 2)... -Execution abc123-def456: completed - fetch-data: completed - summarize: completed -``` - -If the workflow references a credential and you have `MANTLE_ENCRYPTION_KEY` configured, the engine uses the Postgres-backed credential resolver. Without the encryption key, credentials fall back to environment variables only. - -**Errors:** - -If the workflow has not been applied: - -``` -Error: workflow "my-workflow" not found — have you run 'mantle apply'? -``` - ---- - -### mantle cancel - -Cancel a running or pending workflow execution. Marks the execution and any in-progress steps as cancelled. - -``` -Usage: - mantle cancel -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `execution-id` | Yes | UUID of the execution to cancel. | - -**Example:** - -```bash -$ mantle cancel abc123-def456 -Cancelled execution abc123-def456 -``` - -If the execution has already finished: - -```bash -$ mantle cancel abc123-def456 -Execution abc123-def456 is already completed -``` - ---- - -### mantle logs - -View execution logs. When called with an execution ID, shows step-by-step detail. When called without arguments, lists recent executions with optional filters. - -``` -Usage: - mantle logs [execution-id] [flags] -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `execution-id` | No | UUID of a specific execution. When omitted, lists recent executions. | - -**Flags (list mode only):** - -| Flag | Default | Description | -|---|---|---| -| `--workflow` | -- | Filter by workflow name. | -| `--status` | -- | Filter by status: `pending`, `running`, `completed`, `failed`, `cancelled`. | -| `--since` | -- | Show executions started within this duration. Accepts Go durations (`1h`, `30m`) and day notation (`7d`). | -| `--limit` | `20` | Maximum number of executions to show. | - -**Example -- detail mode (with execution ID):** - -```bash -$ mantle logs abc123-def456 -Execution: abc123-def456 -Workflow: fetch-and-summarize (version 2) -Status: completed -Started: 2026-03-18T14:30:00Z -Completed: 2026-03-18T14:30:05Z -Duration: 5.123s - -Steps: - fetch-data completed (1.2s) - summarize completed (3.9s) -``` - -If a step failed, the error is shown below the step: - -``` -Steps: - fetch-data failed (0.5s) - error: ai/completion: API returned 401: Unauthorized -``` - -**Example -- list mode (without arguments):** - -```bash -$ mantle logs -ID WORKFLOW VERSION STATUS STARTED COMPLETED ----------------------------------------------------------------------------------------------------------------------------- -a1b2c3d4-e5f6-7890-abcd-ef1234567890 fetch-and-summarize 2 completed 2026-03-18 14:30:00 2026-03-18 14:30:05 -b2c3d4e5-f6a7-8901-bcde-f12345678901 hello-world 1 completed 2026-03-18 14:28:00 2026-03-18 14:28:01 - -2 execution(s) shown. -``` - -**Example -- filtered list:** - -```bash -$ mantle logs --workflow hello-world --status failed --since 24h --limit 10 -``` - ---- - -### mantle status - -View the current state of a workflow execution with a summary of step statuses. - -``` -Usage: - mantle status -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `execution-id` | Yes | UUID of the execution. | - -**Example:** - -```bash -$ mantle status abc123-def456 -Execution: abc123-def456 -Workflow: fetch-and-summarize (version 2) -Status: completed -Started: 2026-03-18T14:30:00Z -Completed: 2026-03-18T14:30:05Z - -Steps: - completed: 2 -``` - ---- - -### mantle secrets create - -Create a new encrypted credential. Requires `MANTLE_ENCRYPTION_KEY` or `encryption.key` to be configured. - -``` -Usage: - mantle secrets create [flags] -``` - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--name` | Yes | Credential name. Used to reference the credential in workflow steps. | -| `--type` | Yes | Credential type: `generic`, `bearer`, `openai`, `basic`. | -| `--field KEY=VALUE` | Yes | Field value. Repeat for each field the credential type requires. | - -**Example:** - -```bash -$ mantle secrets create --name my-openai --type openai \ - --field api_key=sk-proj-abc123 \ - --field org_id=org-xyz789 -Created credential "my-openai" (type: openai) -``` - -See the [Secrets Guide](secrets-guide.md) for credential types, required fields, and usage in workflows. - ---- - -### mantle secrets list - -List all stored credentials. Shows name, type, and creation date. Never displays decrypted values. - -``` -Usage: - mantle secrets list -``` - -**Example:** - -```bash -$ mantle secrets list -NAME TYPE CREATED -my-openai openai 2026-03-18 14:30:00 -my-api basic 2026-03-18 14:35:00 -``` - ---- - -### mantle secrets delete - -Permanently delete a credential by name. - -``` -Usage: - mantle secrets delete [flags] -``` - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--name` | Yes | Name of the credential to delete. | - -**Example:** - -```bash -$ mantle secrets delete --name my-openai -Deleted credential "my-openai" -``` - ---- - -### mantle secrets rotate-key - -Re-encrypt all stored credentials with a new master key. Use this for key rotation after a security incident or as part of a periodic rotation policy. - -``` -Usage: - mantle secrets rotate-key [flags] -``` - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--new-key` | No | Hex-encoded 32-byte new encryption key. If omitted, a new key is auto-generated. | - -**Example:** - -```bash -$ mantle secrets rotate-key -Re-encrypted 3 credential(s). -New key: a1b2c3d4... -Update MANTLE_ENCRYPTION_KEY to the new key and restart. -``` - -After rotating, update `MANTLE_ENCRYPTION_KEY` (or `encryption.key` in `mantle.yaml`) to the new key before running any other commands. - ---- - -### mantle serve - -Start Mantle as a persistent server with an HTTP API, cron scheduler, and webhook listener. This is the primary way to run Mantle in production. - -``` -Usage: - mantle serve [flags] -``` - -**Flags:** - -| Flag | Env Var | Default | Description | -|---|---|---|---| -| `--api-address` | `MANTLE_API_ADDRESS` | `:8080` | Listen address for the HTTP server. | - -**Behavior:** - -When you run `mantle serve`, Mantle: - -1. Runs all pending database migrations automatically -2. Starts the HTTP API on the configured address -3. Starts the cron scheduler, polling every 30 seconds for due triggers -4. Registers webhook listener endpoints for all applied workflows with webhook triggers -5. Serves health endpoints at `/healthz` and `/readyz` - -**Example:** - -```bash -$ mantle serve -Running migrations... -Migrations complete. -Starting server on :8080 -Cron scheduler started (poll interval: 30s) -``` - -**Custom address:** - -```bash -$ mantle serve --api-address :9090 -Starting server on :9090 -``` - -**Graceful shutdown:** - -Mantle shuts down gracefully on `SIGTERM` or `SIGINT`. When the signal is received: - -1. The HTTP server stops accepting new connections -2. In-flight requests are allowed to complete -3. Running workflow executions finish their current step and checkpoint -4. The process exits with code 0 - -```bash -$ mantle serve -Starting server on :8080 -^C -Shutting down gracefully... -Server stopped. -``` - -**REST API endpoints:** - -The server exposes these API endpoints: - -| Method | Path | Description | -|---|---|---| -| `POST` | `/api/v1/run/{workflow}` | Trigger a workflow execution. Returns the execution ID, workflow name, and version. | -| `POST` | `/api/v1/cancel/{execution}` | Cancel a running execution. | -| `POST` | `/hooks/` | Webhook trigger endpoint. The request body is available as `trigger.payload` in the workflow's inputs. | -| `GET` | `/healthz` | Liveness probe. Returns 200 when the process is running. | -| `GET` | `/readyz` | Readiness probe. Returns 200 when the database connection is healthy and migrations are applied. | - -**Example -- trigger a workflow via the API:** - -```bash -$ curl -s -X POST http://localhost:8080/api/v1/run/fetch-and-summarize | jq . -{ - "execution_id": "abc123-def456", - "workflow": "fetch-and-summarize", - "version": 2 -} -``` - -**Example -- cancel an execution via the API:** - -```bash -$ curl -s -X POST http://localhost:8080/api/v1/cancel/abc123-def456 -``` - -**Example -- trigger a webhook:** - -```bash -$ curl -s -X POST http://localhost:8080/hooks/my-workflow \ - -H "Content-Type: application/json" \ - -d '{"event": "deploy", "repo": "my-app"}' -``` - -Requires a database connection. See the [Server Guide](server-guide.md) for production deployment guidance. - ---- - -### mantle audit - -Query the immutable audit trail. Every state-changing operation in Mantle emits an audit event to Postgres. This command queries those events with optional filters. - -``` -Usage: - mantle audit [flags] -``` - -**Flags:** - -| Flag | Default | Description | -|---|---|---| -| `--action` | -- | Filter by action type (e.g., `workflow.applied`, `step.completed`, `execution.cancelled`). | -| `--actor` | -- | Filter by actor (e.g., `cli`, `engine`, a user ID). | -| `--resource` | -- | Filter by resource as `type/id` (e.g., `workflow_definition/my-workflow`). | -| `--since` | -- | Show events within this duration. Accepts Go durations (`1h`, `30m`) and day notation (`7d`). | -| `--limit` | `50` | Maximum number of events to show. | - -**Action Types:** - -| Action | Description | -|---|---| -| `workflow.applied` | A workflow definition was applied (new version stored). | -| `workflow.executed` | A workflow execution started. | -| `step.started` | A step began executing. | -| `step.completed` | A step finished successfully. | -| `step.failed` | A step failed. | -| `step.skipped` | A step was skipped (due to an `if` condition evaluating to `false`). | -| `execution.cancelled` | An execution was cancelled. | - -**Example -- all recent events:** - -```bash -$ mantle audit -2026-03-18T14:30:00Z cli workflow.applied workflow_definition/hello-world -2026-03-18T14:30:01Z engine workflow.executed workflow_execution/a1b2c3d4 -2026-03-18T14:30:01Z engine step.started step_execution/fetch -2026-03-18T14:30:02Z engine step.completed step_execution/fetch -``` - -**Example -- filtered:** - -```bash -$ mantle audit --action workflow.applied --since 7d --limit 20 -$ mantle audit --actor cli --resource workflow_definition/hello-world -``` - ---- - -### mantle plugins - -Manage third-party connector plugins. Plugins are executable binaries that extend Mantle with custom connector actions. - -``` -Usage: - mantle plugins list - mantle plugins install - mantle plugins remove -``` - -#### mantle plugins list - -List all installed plugins in the plugin directory (`.mantle/plugins/` by default). - -```bash -$ mantle plugins list -my-custom-connector .mantle/plugins/my-custom-connector -``` - -If no plugins are installed: - -``` -(no plugins installed) -``` - -#### mantle plugins install - -Install a plugin by copying the binary into the plugin directory. - -```bash -$ mantle plugins install ./build/my-custom-connector -Installed plugin from ./build/my-custom-connector -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `path` | Yes | Path to the plugin binary to install. | - -#### mantle plugins remove - -Remove an installed plugin by name. - -```bash -$ mantle plugins remove my-custom-connector -Removed plugin my-custom-connector -``` - -**Arguments:** - -| Argument | Required | Description | -|---|---|---| -| `name` | Yes | Name of the plugin to remove (the filename in the plugins directory). | - -See the [Plugins Guide](plugins-guide.md) for how to write and test a plugin. - ---- - -### mantle teams - -Manage teams. Teams are the unit of multi-tenancy in Mantle -- each team has its own workflows, credentials, and users. - -``` -Usage: - mantle teams create [flags] - mantle teams list - mantle teams delete [flags] -``` - -#### mantle teams create - -Create a new team. - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--name` | Yes | Team name. | - -**Example:** - -```bash -$ mantle teams create --name my-team -Created team my-team (id: a1b2c3d4-e5f6-7890-abcd-ef1234567890) -``` - -#### mantle teams list - -List all teams. - -```bash -$ mantle teams list -NAME ID CREATED -my-team a1b2c3d4-e5f6-7890-abcd-ef1234567890 2026-03-18 14:30:00 -default b2c3d4e5-f6a7-8901-bcde-f12345678901 2026-03-18 10:00:00 -``` - -If no teams exist: - -``` -(no teams) -``` - -#### mantle teams delete - -Delete a team by name. - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--name` | Yes | Name of the team to delete. | - -**Example:** - -```bash -$ mantle teams delete --name my-team -Deleted team "my-team" -``` - ---- - -### mantle users - -Manage users. Users belong to teams and have a role that controls their permissions. - -``` -Usage: - mantle users create [flags] - mantle users list [flags] - mantle users delete [flags] - mantle users api-key [flags] -``` - -#### mantle users create - -Create a new user and assign them to a team with a role. - -**Flags:** - -| Flag | Required | Default | Description | -|---|---|---|---| -| `--email` | Yes | -- | User email address. | -| `--name` | Yes | -- | User display name. | -| `--team` | No | `default` | Team to add the user to. | -| `--role` | No | `operator` | Role to assign: `admin`, `team_owner`, `operator`. | - -**Example:** - -```bash -$ mantle users create --email alice@example.com --name "Alice Smith" --team my-team --role team_owner -Created user alice@example.com (role: team_owner, team: my-team) -``` - -#### mantle users list - -List users in a team. - -**Flags:** - -| Flag | Required | Default | Description | -|---|---|---|---| -| `--team` | No | `default` | Team name to list users for. | - -**Example:** - -```bash -$ mantle users list --team my-team -EMAIL NAME ROLE -alice@example.com Alice Smith team_owner -bob@example.com Bob Jones operator -``` - -If no users exist: - -``` -(no users) -``` - -#### mantle users delete - -Delete a user by email. - -**Flags:** - -| Flag | Required | Default | Description | -|---|---|---|---| -| `--email` | Yes | -- | Email of the user to delete. | -| `--team` | No | `default` | Team name. | - -**Example:** - -```bash -$ mantle users delete --email bob@example.com -Deleted user "bob@example.com" -``` - -#### mantle users api-key - -Generate an API key for a user. The key is displayed once and cannot be retrieved again. - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--email` | Yes | User email. | -| `--key-name` | Yes | A name for the API key (for identification). | - -**Example:** - -```bash -$ mantle users api-key --email alice@example.com --key-name ci-deploy - -API Key: mk_a1b2c3d4e5f6... - -Save this key — it cannot be retrieved again. -Key prefix for reference: mk_a1b2 -``` - ---- - -### mantle roles - -Manage user roles. - -``` -Usage: - mantle roles assign [flags] -``` - -#### mantle roles assign - -Assign a role to an existing user. - -**Flags:** - -| Flag | Required | Default | Description | -|---|---|---|---| -| `--email` | Yes | -- | User email. | -| `--role` | Yes | -- | Role to assign: `admin`, `team_owner`, `operator`. | -| `--team` | No | `default` | Team name. | - -**Example:** - -```bash -$ mantle roles assign --email alice@example.com --role admin -Assigned role "admin" to user "alice@example.com" -``` - ---- - -### mantle login - -Authenticate with a Mantle server. Supports three authentication methods: OIDC authorization code with PKCE (default), device authorization flow, and API key caching. - -Credentials are stored in `~/.mantle/credentials`. - -``` -Usage: - mantle login [flags] -``` - -**Flags:** - -| Flag | Description | -|---|---| -| `--api-key` | Authenticate by entering and caching an API key. | -| `--device` | Use the device authorization flow (for headless/SSH environments). | - -When neither flag is provided, the default OIDC authorization code + PKCE flow is used. This opens a browser for the identity provider login and listens on a local callback URL. - -**Example -- OIDC (default):** - -```bash -$ mantle login -Open this URL to authenticate: - - https://auth.example.com/authorize?client_id=... - -Waiting for callback... -Login successful! Credentials saved to /home/alice/.mantle/credentials -``` - -**Example -- device flow:** - -```bash -$ mantle login --device -To authenticate, visit: - - https://auth.example.com/device - -And enter code: ABCD-1234 - -Waiting for authorization... -Login successful! Credentials saved to /home/alice/.mantle/credentials -``` - -**Example -- API key:** - -```bash -$ mantle login --api-key -Enter API key: mk_a1b2c3d4e5f6... -API key saved to /home/alice/.mantle/credentials -``` - -OIDC requires `auth.oidc.issuer_url` and `auth.oidc.client_id` to be configured in `mantle.yaml` or via environment variables. - ---- - -### mantle logout - -Remove cached credentials from `~/.mantle/credentials`. - -``` -Usage: - mantle logout -``` - -**Example:** - -```bash -$ mantle logout -Credentials removed from /home/alice/.mantle/credentials -``` - ---- - -### mantle cleanup - -Remove old execution data and audit events based on a retention policy. Uses flag values or falls back to the `retention` section in `mantle.yaml`. - -``` -Usage: - mantle cleanup [flags] -``` - -**Flags:** - -| Flag | Default | Description | -|---|---|---| -| `--execution-days` | `0` (use config) | Delete workflow executions older than N days. | -| `--audit-days` | `0` (use config) | Delete audit events older than N days. | - -If neither flag is set and no retention config exists, the command does nothing. - -**Example:** - -```bash -$ mantle cleanup --execution-days 30 --audit-days 90 -Deleted 42 workflow execution(s) older than 30 day(s). -Deleted 156 audit event(s) older than 90 day(s). -``` - -**Example -- using config defaults:** - -```bash -$ mantle cleanup -Deleted 12 workflow execution(s) older than 30 day(s). -Deleted 89 audit event(s) older than 90 day(s). -``` - ---- - -### mantle library - -Manage the shared workflow template library. Templates let you publish reusable workflow definitions that other teams can deploy. - -``` -Usage: - mantle library publish [flags] - mantle library list - mantle library deploy [flags] -``` - -#### mantle library publish - -Publish a workflow as a shared template. Reads the latest applied version of the named workflow and stores it in the shared library. If a template with the same name already exists, it is updated. - -```bash -$ mantle library publish --workflow daily-report -Published "daily-report" to shared library -``` - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--workflow` | Yes | Name of the applied workflow to publish. | - -#### mantle library list - -List all shared workflow templates. - -```bash -$ mantle library list -NAME DESCRIPTION -daily-report Generate and email a daily summary report -api-monitor Hourly API health check with Slack alerts -``` - -If no templates exist: - -``` -(no templates) -``` - -#### mantle library deploy - -Deploy a shared template as a workflow definition in the target team. Creates a new version in the workflow_definitions table. - -```bash -$ mantle library deploy --template daily-report -Deployed "daily-report" as version 1 -``` - -**Flags:** - -| Flag | Required | Description | -|---|---|---| -| `--template` | Yes | Name of the template to deploy. | -| `--team` | No | Target team ID. Defaults to the default team. | - ---- - -## REST API - -The `mantle serve` command exposes a REST API for programmatic access to Mantle. All endpoints return JSON. - -### Endpoints - -| Method | Path | Description | -|---|---|---| -| `POST` | `/api/v1/run/{workflow}` | Trigger a workflow execution. | -| `POST` | `/api/v1/cancel/{execution}` | Cancel a running execution. | -| `GET` | `/api/v1/executions` | List executions with optional filters. | -| `GET` | `/api/v1/executions/{id}` | Get execution details with step information. | -| `POST` | `/hooks/{path}` | Webhook trigger endpoint. | -| `GET` | `/healthz` | Liveness probe. | -| `GET` | `/readyz` | Readiness probe (checks database connectivity). | -| `GET` | `/metrics` | Prometheus metrics endpoint. | - -### POST /api/v1/run/{workflow} - -Triggers a new execution of the named workflow using the latest applied version. - -```bash -curl -s -X POST http://localhost:8080/api/v1/run/fetch-and-summarize | jq . -``` - -```json -{ - "execution_id": "abc123-def456", - "workflow": "fetch-and-summarize", - "version": 2 -} -``` - -### POST /api/v1/cancel/{execution} - -Cancels a running or pending execution. - -```bash -curl -s -X POST http://localhost:8080/api/v1/cancel/abc123-def456 -``` - -```json -{ - "execution_id": "abc123-def456", - "status": "cancelled" -} -``` - -### GET /api/v1/executions - -Lists recent executions. Supports query parameters for filtering. - -**Query Parameters:** - -| Parameter | Type | Description | -|---|---|---| -| `workflow` | string | Filter by workflow name. | -| `status` | string | Filter by status: `pending`, `running`, `completed`, `failed`, `cancelled`. | -| `since` | string | Show executions within this duration (e.g., `1h`, `24h`, `7d`). | -| `limit` | number | Maximum results. Default: `20`. | - -```bash -curl -s "http://localhost:8080/api/v1/executions?workflow=hello-world&status=completed&limit=5" | jq . -``` - -```json -{ - "executions": [ - { - "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "workflow": "hello-world", - "version": 1, - "status": "completed", - "started_at": "2026-03-18T14:30:00Z", - "completed_at": "2026-03-18T14:30:01Z" - } - ] -} -``` - -### GET /api/v1/executions/{id} - -Returns detailed information about a single execution, including all step results. - -```bash -curl -s http://localhost:8080/api/v1/executions/a1b2c3d4-e5f6-7890-abcd-ef1234567890 | jq . -``` - -```json -{ - "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "workflow": "hello-world", - "version": 1, - "status": "completed", - "started_at": "2026-03-18T14:30:00Z", - "completed_at": "2026-03-18T14:30:01Z", - "steps": [ - { - "name": "fetch", - "status": "completed", - "started_at": "2026-03-18T14:30:00Z", - "completed_at": "2026-03-18T14:30:01Z" - } - ] -} -``` - ---- - -## Exit Codes - -| Code | Meaning | -|---|---| -| `0` | Success | -| `1` | Validation error, runtime error, or command failure | diff --git a/docs/comparison.md b/docs/comparison.md deleted file mode 100644 index cd7799a..0000000 --- a/docs/comparison.md +++ /dev/null @@ -1,66 +0,0 @@ -# How Mantle Compares - -Mantle occupies a specific niche: headless, IaC-first AI workflow automation. Several excellent tools overlap with parts of what Mantle does. This document provides an honest comparison to help you decide whether Mantle is the right fit for your use case. - -## Mantle vs Temporal - -[Temporal](https://temporal.io/) is a durable execution platform that guarantees workflow completion even through infrastructure failures. It powers mission-critical systems at companies like Uber, Netflix, and Snap. - -Both Temporal and Mantle provide durable workflow execution with checkpoint-and-resume semantics. Both persist workflow state so that a crash mid-execution does not lose progress. The overlap is real, and if your primary concern is bulletproof transaction orchestration across microservices, Temporal is the more mature choice. - -The differences are in operational complexity and target audience. Temporal requires running a multi-service cluster (frontend, history, matching, and worker services) plus a database backend. Your workflow logic is written in Go or Java using the Temporal SDK, which gives you enormous flexibility but also means your workflows are compiled code that needs to be deployed as services. Mantle is a single binary that reads YAML workflow definitions and connects to Postgres. There is no SDK, no worker fleet, no cluster topology to manage. - -Mantle also has first-class AI/LLM support that Temporal does not. Mantle's `ai/completion` connector handles model routing, structured output, tool use loops, and secrets management for API keys out of the box. In Temporal, you would build all of this as activity implementations in your worker code. - -**Choose Temporal** when you need battle-tested distributed transaction orchestration, saga patterns, or are already running a microservices architecture with dedicated platform teams. **Choose Mantle** when you want to define AI-powered workflows in YAML, version them in git, and deploy with a single binary. - -## Mantle vs n8n / Zapier - -[n8n](https://n8n.io/) and [Zapier](https://zapier.com/) are workflow automation platforms built around visual, drag-and-drop interfaces. They excel at connecting SaaS applications without writing code. - -n8n and Zapier are outstanding tools for non-technical users who need to wire together integrations quickly. The visual canvas makes it easy to see the flow of data, and the library of pre-built integrations covers hundreds of SaaS products. If your team consists of business analysts or operations staff who are not comfortable with YAML and git, these tools are almost certainly a better fit than Mantle. - -Mantle takes the opposite approach: workflows are YAML files that live in git repositories. Changes go through the same code review process as application code. The `validate` / `plan` / `apply` lifecycle gives you the same confidence you get from Terraform: you know exactly what will change before it changes. This is a significant advantage for platform engineering teams that already use IaC practices and want workflow definitions to be part of their CI/CD pipeline. - -n8n is self-hostable and open source, which makes it the closer comparison. The key difference is that n8n stores workflow definitions in its own database and exposes them through a web UI, while Mantle treats workflow definitions as source code artifacts. n8n also has a much larger connector ecosystem today; Mantle ships with HTTP and AI connectors and relies on its gRPC plugin protocol for extension. - -**Choose n8n or Zapier** when your users are non-technical, when you need a visual workflow builder, or when you need hundreds of pre-built SaaS integrations. **Choose Mantle** when you want version-controlled workflow definitions, CI/CD-driven deployment, and first-class AI/LLM capabilities. - -## Mantle vs LangChain / CrewAI - -[LangChain](https://www.langchain.com/) and [CrewAI](https://www.crewai.com/) are Python frameworks for building applications that use large language models. LangChain provides composable abstractions for chains, agents, and tool use. CrewAI builds on top of LangChain to orchestrate multi-agent collaboration. - -These are excellent tools for prototyping and building LLM-powered applications in Python. LangChain's ecosystem is enormous: it has integrations with dozens of model providers, vector databases, and retrieval systems. If you are a Python developer building a custom AI application, LangChain gives you the most flexibility and the largest community. - -The distinction is between a library and a platform. LangChain is code that runs inside your application. You are responsible for deployment, scaling, error handling, state persistence, secrets management, and observability. Mantle is an execution platform that handles all of these concerns. When a Mantle workflow step fails, the engine records the checkpoint, applies your retry policy, and resumes from the last completed step on restart. When a LangChain chain fails, your application code needs to handle that. - -Mantle is also language-agnostic. Workflow definitions are YAML with CEL expressions, so your team does not need Python expertise to define and operate AI workflows. This matters in organizations where the platform team operates the automation infrastructure but does not want to own Python application code. - -**Choose LangChain or CrewAI** when you are building a custom AI application in Python, need fine-grained control over prompting and retrieval, or want access to the largest LLM framework ecosystem. **Choose Mantle** when you want a managed execution platform with checkpointing, secrets, audit trails, and a declarative workflow format that does not require writing application code. - -## Mantle vs Prefect / Airflow - -[Prefect](https://www.prefect.io/) and [Apache Airflow](https://airflow.apache.org/) are workflow orchestration platforms designed primarily for data engineering. Airflow pioneered the concept of DAGs-as-code for scheduling ETL pipelines. Prefect modernized the pattern with a more Pythonic API and better developer experience. - -Both tools are proven at scale for data pipeline orchestration. Airflow has an enormous user base and plugin ecosystem. Prefect offers a more modern approach with dynamic workflows, better error handling, and a hybrid execution model. If your primary use case is orchestrating Python data pipelines, ETL jobs, or dbt runs, these tools have years of battle-testing and community support that Mantle cannot match. - -The workflows in Prefect and Airflow are Python functions decorated with framework-specific annotations. This is powerful for data teams that already work in Python, but it means your workflow definitions are tightly coupled to the Python ecosystem. Mantle's YAML-based definitions are language-agnostic and can be authored, reviewed, and deployed without a Python runtime. - -Mantle's key differentiator here is first-class AI/LLM support. Prefect and Airflow can invoke LLM APIs through custom tasks, but they have no built-in concepts for model routing, structured output schemas, tool use loops, or credential management for AI providers. Mantle's `ai/completion` connector handles these concerns natively, making it better suited for workflows where AI processing is a core part of the pipeline rather than an afterthought. - -**Choose Prefect or Airflow** when you are running Python data pipelines, need mature scheduling and dependency management for ETL/ELT, or have an established data engineering team. **Choose Mantle** when your workflows are AI-centric, you want a language-agnostic declarative format, or you prefer a single-binary deployment model over a Python-based platform. - -## Summary Comparison - -| | Mantle | Temporal | n8n / Zapier | LangChain / CrewAI | Prefect / Airflow | -|---|---|---|---|---|---| -| **Primary use case** | AI workflow automation | Distributed transactions | SaaS integration | LLM application development | Data pipelines | -| **Workflow format** | YAML + CEL | Go / Java SDK | Visual canvas | Python code | Python code | -| **Deployment** | Single binary + Postgres | Multi-service cluster | Self-hosted or cloud | Library in your app | Python platform | -| **AI/LLM support** | First-class (built-in) | Build it yourself | Partial (AI nodes) | First-class (library) | Build it yourself | -| **Checkpointing** | Built-in | Built-in | Partial | None (you build it) | Built-in | -| **Secrets management** | Built-in, encrypted | External | Built-in | External | External | -| **Version control** | Git-native (IaC) | Code in repos | Database-stored | Code in repos | Code in repos | -| **Target user** | Platform engineers | Backend engineers | Non-technical users | Python developers | Data engineers | -| **Operational complexity** | Low | High | Medium | N/A (library) | Medium-High | -| **Ecosystem maturity** | Early | Mature | Mature | Mature | Mature | diff --git a/docs/concepts.md b/docs/concepts.md deleted file mode 100644 index 315064f..0000000 --- a/docs/concepts.md +++ /dev/null @@ -1,576 +0,0 @@ -# Concepts - -This page explains the core ideas behind Mantle's design: why workflows are treated as infrastructure, how versioning works, what checkpointing guarantees (and does not guarantee), and how data flows between steps. - -## Infrastructure as Code Lifecycle - -Mantle borrows the validate-plan-apply pattern from infrastructure tools like Terraform. Workflow definitions are not executed directly from YAML files. Instead, they go through a controlled deployment lifecycle: - -``` -mantle validate --> mantle plan --> mantle apply --> mantle run - (offline) (diff against (store new (execute - database) version) latest) -``` - -**validate** parses the YAML and checks it against structural rules: naming conventions, required fields, valid types, and correct durations. This runs offline with no database connection, so you can run it in a pre-commit hook or CI pipeline before anything touches the database. - -**plan** compares your local file against the latest version stored in the database and shows a diff of what will change. Nothing is written. - -**apply** validates the workflow, hashes the content with SHA-256, compares the hash against the latest stored version, and -- if the content changed -- inserts a new immutable version into the `workflow_definitions` table. If nothing changed, it reports "No changes" and does nothing. - -**run** executes the latest applied version of a workflow, checkpointing each step to Postgres as it completes. - -This lifecycle has a few important properties: - -- **Every version is immutable.** Once applied, a version is never modified or deleted. Version 1 of a workflow always contains exactly what was applied as version 1. -- **Deployments are auditable.** You can trace what definition was active at any point in time by looking at version numbers and timestamps. -- **Validation is separated from storage.** You can validate dozens of files in CI without ever connecting to a database. - -## Versioned Definitions - -Every time you `mantle apply` a workflow with changed content, Mantle creates a new version with an incremented version number. The version history is strictly append-only. - -``` -mantle apply workflow.yaml # Creates version 1 -# edit workflow.yaml -mantle apply workflow.yaml # Creates version 2 -mantle apply workflow.yaml # No changes — still version 2 -# edit workflow.yaml again -mantle apply workflow.yaml # Creates version 3 -``` - -Mantle determines whether content has changed by comparing SHA-256 hashes of the raw YAML file content. If the hash matches the latest version, no new version is created. This means whitespace-only changes or comment changes do create new versions (since the raw bytes change), while applying the same file twice does not. - -Each version record in the database stores: - -| Column | Description | -|---|---| -| `id` | Unique UUID | -| `name` | Workflow name (e.g., `fetch-and-summarize`) | -| `version` | Integer version number, starting at 1 | -| `content` | The parsed workflow definition as JSON | -| `content_hash` | SHA-256 hash of the raw YAML file | -| `created_at` | Timestamp of when this version was applied | - -## Checkpoint and Resume - -When Mantle executes a workflow, each step's result is checkpointed to Postgres before the next step begins. If the process crashes mid-execution, it resumes from the last completed step rather than starting over. - -### What Checkpointing Guarantees - -- **No duplicate internal work.** A step that completed and was checkpointed before a crash is not re-executed after recovery. -- **Durable state.** Step outputs survive process restarts because they are stored in Postgres, not in memory. -- **Crash recovery.** A workflow execution can be resumed by any Mantle instance with access to the same database. - -### What Checkpointing Does Not Guarantee - -- **Exactly-once external side effects.** If a step makes an HTTP POST and the process crashes after the POST completes but before the checkpoint is written, the POST will be repeated on recovery. This is inherent to any system that interacts with external services. Use idempotency keys in your external APIs to handle this. -- **Atomicity across steps.** Each step is independent. There is no rollback of previously completed steps if a later step fails. - -The database schema for execution tracking uses three tables: - -- `workflow_executions` -- one row per workflow run, tracking overall status (`pending`, `running`, `completed`, `failed`, `cancelled`) -- `step_executions` -- one row per step attempt, tracking status, output, and errors -- Each step attempt is uniquely identified by `(execution_id, step_name, attempt)`, supporting retries - -## CEL Expressions and Data Flow - -Data flows between steps through [CEL (Common Expression Language)](https://github.com/google/cel-go) expressions. CEL is a small, fast, non-Turing-complete expression language designed by Google for security and policy evaluation. - -### Three Namespaces - -| Namespace | Example | Description | -|---|---|---| -| `inputs` | `inputs.url` | Values passed when the workflow is triggered. | -| `steps` | `steps.fetch-data.output.body` | Output from a previously completed step. | -| `env` | `env.API_TOKEN` | Environment variables available to the engine. | - -### Where CEL Appears - -**Conditional execution** -- the `if` field on a step: - -```yaml -if: "steps.fetch-data.output.status_code == 200" -``` - -The step runs only when this expression evaluates to `true`. If the expression evaluates to `false`, the step is skipped. - -**Template interpolation** -- double-brace syntax in `params`: - -```yaml -params: - url: "{{ inputs.url }}" - prompt: "Summarize: {{ steps.fetch-data.output.body }}" -``` - -Template expressions are evaluated and their results are substituted into the string. - -### Data Flow Example - -Consider this workflow: - -```yaml -inputs: - url: - type: string - -steps: - - name: fetch-data - action: http/request - params: - method: GET - url: "{{ inputs.url }}" - - - name: summarize - action: ai/completion - params: - provider: openai - model: gpt-4o - prompt: "Summarize: {{ steps.fetch-data.output.body }}" -``` - -The data flows like this: - -1. The caller provides `url` as an input when triggering the workflow. -2. Step `fetch-data` reads `inputs.url` and makes an HTTP GET request. -3. The HTTP connector returns output with fields like `status_code`, `headers`, and `body`. -4. Step `summarize` reads `steps.fetch-data.output.body` to build its prompt. -5. The AI connector returns the completion result. - -Each step can only reference outputs from steps that have completed before it runs. The engine detects these references automatically and treats them as implicit dependencies. When combined with explicit `depends_on` declarations, this enables parallel execution -- see [Parallel Execution](#parallel-execution) below. - -## Parallel Execution - -Mantle does not execute steps strictly in list order. Instead, it builds a directed acyclic graph (DAG) from both explicit `depends_on` declarations and implicit dependencies detected from CEL expression analysis (e.g., `steps.fetch-data.output`), then schedules steps for concurrent execution when their dependencies allow it. - -**How it works:** - -1. Steps with no dependencies start immediately and run in parallel. -2. When a step completes (or is skipped), the engine checks which downstream steps now have all dependencies resolved and starts them. -3. If a step fails, all downstream steps that transitively depend on it are cancelled. - -**Implicit dependency detection:** When you reference `steps['fetch-data'].output.body` in a step's `params` or `if` field, the engine automatically adds `fetch-data` as a dependency. You do not need to redundantly list it in `depends_on`. - -**Skipped steps count as resolved.** If a step's `if` condition evaluates to `false`, it is marked as skipped. Downstream steps that depend on it are still unblocked -- they can proceed, though referencing the skipped step's output will yield an empty value. - -For the full `depends_on` field reference and a fan-out/fan-in YAML example, see the [Workflow Reference](workflow-reference.md#parallel-execution). - -## AI Tool Use - -The `ai/completion` connector supports multi-turn tool use (function calling). You declare tools in the step's `params`, each mapping a tool name to a Mantle connector action. The engine then orchestrates a loop between the LLM and your tools: - -1. The engine sends the prompt (and tool definitions) to the LLM. -2. The LLM responds with either a text completion or one or more tool call requests. -3. For each tool call, the engine executes the corresponding connector action and collects the result. -4. The tool results are appended to the conversation and sent back to the LLM. -5. Steps 2-4 repeat until the LLM produces a final text response, or the configured round limit is reached. - -**Safety limits:** The `max_tool_rounds` param (default: 10) caps the number of LLM-tool round trips. The `max_tool_calls_per_round` param (default: 10) caps how many tools the LLM can invoke in a single turn. If the round limit is exhausted, the engine makes one final call asking the LLM to respond with the information gathered so far. - -**Error handling:** If a tool execution fails, the error message is sent back to the LLM as the tool result rather than crashing the workflow. This gives the LLM the opportunity to retry with different arguments or proceed without that tool's output. - -See the [Workflow Reference](workflow-reference.md#tool-declarations) for the tool schema and a complete YAML example. - -## Connectors - -Connectors are the integration points between Mantle and external systems. Each connector provides one or more actions that steps can invoke. - -### HTTP Connector - -The `http/request` action makes HTTP requests. It is the general-purpose connector for interacting with REST APIs, webhooks, and any HTTP endpoint. - -Key design points: - -- JSON request bodies are automatically serialized -- JSON response bodies are automatically parsed into structured data accessible via CEL -- You control headers, method, URL, and body through step params - -See the [Workflow Reference](workflow-reference.md#httprequest) for the complete parameter and output specification. - -### AI Connector - -The `ai/completion` action sends prompts to OpenAI-compatible chat completion APIs. - -Key design points: - -- **BYOK (Bring Your Own Key)** -- Mantle does not proxy through a hosted service. You provide your own API keys through the secrets system and reference them with the `credential` field on your workflow step. -- **Structured output** -- you can pass an `output_schema` parameter with a JSON Schema, and the model returns JSON conforming to that schema. The parsed result is available as `steps.STEP_NAME.output.json`. -- **Custom endpoints** -- the `base_url` parameter lets you point to any OpenAI-compatible API (Azure OpenAI, Ollama, vLLM, etc.) instead of the default `https://api.openai.com/v1`. -- **Tool use (function calling)** -- you can declare `tools` on an AI step, each mapping to a Mantle connector action. The engine runs a multi-turn loop: the LLM requests tool calls, the engine executes them via connectors, feeds results back, and repeats until the LLM produces a final text response or configured limits (`max_tool_rounds`, `max_tool_calls_per_round`) are reached. - -See the [Workflow Reference](workflow-reference.md#aicompletion) for the complete parameter and output specification. - -### Slack Connector - -The `slack/send` and `slack/history` actions interact with the Slack Web API. They handle authentication, request formatting, and error parsing for you. - -Use cases: -- Sending notifications to a team channel when a workflow succeeds or fails -- Reading recent messages from a channel to summarize or process - -See the [Workflow Reference](workflow-reference.md#slacksend) for parameters and output. - -### Postgres Connector - -The `postgres/query` action executes parameterized SQL against external Postgres databases. It connects per-step and disconnects afterward, keeping the connector stateless. - -Use cases: -- Reading data from a reporting database to feed into an AI summarization step -- Writing workflow results back to a business database -- Running scheduled data cleanup queries - -See the [Workflow Reference](workflow-reference.md#postgresquery) for parameters and output. - -### Email Connector - -The `email/send` action sends emails via SMTP. It supports plaintext and HTML content, multiple recipients, and configurable SMTP servers. - -See the [Workflow Reference](workflow-reference.md#emailsend) for parameters and output. - -### S3 Connector - -The `s3/put`, `s3/get`, and `s3/list` actions interact with AWS S3 and S3-compatible storage services (MinIO, DigitalOcean Spaces, Backblaze B2). The `endpoint` credential field allows you to point to any S3-compatible API. - -See the [Workflow Reference](workflow-reference.md#s3put) for parameters and output. - -### Connector Summary - -| Action | Description | -|---|---| -| `http/request` | HTTP requests to any URL | -| `ai/completion` | LLM chat completions (OpenAI-compatible) | -| `slack/send` | Send Slack messages | -| `slack/history` | Read Slack channel history | -| `postgres/query` | Execute SQL on external Postgres databases | -| `email/send` | Send email via SMTP | -| `s3/put` | Upload objects to S3 | -| `s3/get` | Download objects from S3 | -| `s3/list` | List objects in S3 | - -## Plugin System - -Plugins extend Mantle with third-party connector actions that run as subprocesses. This keeps the core engine isolated from external code while allowing the connector surface area to grow without modifying the Mantle binary. - -### How Plugins Work - -A plugin is an executable binary that reads JSON from stdin and writes JSON to stdout. The engine invokes the plugin as a subprocess for each step execution, passing the action name, parameters, and credential fields as a JSON payload. - -``` -Engine Plugin Process - | | - |-- spawn subprocess --------->| - |-- write JSON to stdin ------>| - | |-- execute action - |<-- read JSON from stdout ----| - |-- process terminates ------->| -``` - -### Plugin Contract - -The JSON input format: - -```json -{ - "action": "my-plugin/do-thing", - "params": {"key": "value"}, - "credential": {"api_key": "secret"} -} -``` - -The JSON output format: - -```json -{ - "result": "success", - "data": {"processed": true} -} -``` - -If the plugin writes to stderr or exits with a non-zero code, the step fails with the stderr content as the error message. - -### Protobuf Definition - -The plugin contract is formally defined in `proto/connector.proto`. While the current V1.1 implementation uses the simpler JSON stdin/stdout protocol, the protobuf definition serves as the specification for a future gRPC-based plugin protocol. - -The service defines three RPCs: - -- **Execute** -- runs the connector action with parameters and credentials -- **Validate** -- checks whether parameters are valid for this connector -- **Describe** -- returns metadata about the connector's supported actions - -### Plugin Management - -Plugins are stored in the `.mantle/plugins/` directory. Use the CLI to manage them: - -```bash -mantle plugins install ./path/to/my-plugin # Copy binary to plugin directory -mantle plugins list # List installed plugins -mantle plugins remove my-plugin # Remove a plugin -``` - -See the [Plugins Guide](plugins-guide.md) for a complete walkthrough of writing and testing a plugin. - -## Shared Workflow Library - -The workflow library lets teams publish reusable workflow templates and deploy them across environments and teams. This is Mantle's mechanism for sharing best-practice workflows without copy-pasting YAML files. - -### Publish/Deploy Model - -The library uses a two-step model: - -1. **Publish** -- takes a workflow that has been `mantle apply`-ed and stores it as a shared template. The template includes the workflow's name, description, and full definition. - -2. **Deploy** -- copies a shared template into a target team's workflow definitions as a new version. The deployed workflow behaves identically to one created through `mantle apply`. - -``` -Team A: mantle apply daily-report.yaml - mantle library publish --workflow daily-report - -Team B: mantle library list - mantle library deploy --template daily-report -``` - -Publishing the same name again updates the template. Deploying the same template again creates a new version, not a duplicate. - -### When to Use the Library - -- Sharing standard operational workflows (health checks, data syncs) across teams -- Creating starter templates for common patterns (fetch-transform-notify) -- Distributing approved workflows in a multi-tenant environment - -See the [CLI Reference](cli-reference.md#mantle-library) for command details. - -## Observability - -Mantle provides three observability mechanisms: Prometheus metrics, an immutable audit trail, and structured JSON logging. Together, they give you visibility into what your workflows are doing, how they are performing, and who changed what. - -### Prometheus Metrics - -When running in server mode (`mantle serve`), Mantle exposes a `/metrics` endpoint in Prometheus exposition format. Scrape this endpoint with Prometheus, Grafana Agent, or any compatible collector. - -**Exposed metrics:** - -| Metric | Type | Labels | Description | -|---|---|---|---| -| `mantle_workflow_executions_total` | Counter | `workflow`, `status` | Total workflow executions by name and outcome. | -| `mantle_step_executions_total` | Counter | `workflow`, `step`, `status` | Total step executions by workflow, step name, and outcome. | -| `mantle_step_duration_seconds` | Histogram | `workflow`, `step`, `action` | Step execution duration in seconds. | -| `mantle_connector_duration_seconds` | Histogram | `action` | Connector invocation duration in seconds. | -| `mantle_active_executions` | Gauge | -- | Number of currently running workflow executions. | - -### Audit Trail - -Every state-changing operation emits an immutable audit event to the `audit_events` table in Postgres. Events are append-only -- they cannot be modified or deleted. - -Query audit events with the `mantle audit` CLI command. See the [CLI Reference](cli-reference.md#mantle-audit) for filter options. - -### Structured JSON Logging - -In server mode, Mantle emits structured JSON logs to stdout via Go's `slog` package. Each log line is a JSON object with `time`, `level`, `msg`, and contextual fields. - -```json -{"time":"2026-03-18T14:30:00.000Z","level":"INFO","msg":"server listening","address":":8080"} -{"time":"2026-03-18T14:30:01.000Z","level":"INFO","msg":"cron scheduler started"} -{"time":"2026-03-18T14:30:05.000Z","level":"INFO","msg":"workflow execution completed","workflow":"hello-world","execution_id":"abc123"} -``` - -Configure the log level with the `--log-level` flag, `MANTLE_LOG_LEVEL` environment variable, or `log.level` in `mantle.yaml`. Levels: `debug`, `info`, `warn`, `error`. - -The JSON format integrates directly with log aggregation systems like the ELK stack, Datadog, Grafana Loki, and any tool that ingests structured JSON. - -## Secrets and Credential Resolution - -Mantle treats secrets (API keys, tokens, credentials) as opaque handles that are resolved at connector invocation time. You never put raw secret values in workflow YAML. Instead, you create a named credential with `mantle secrets create` and reference it by name in your workflow step's `credential` field. - -### Credential Types - -Each credential has a type that defines its schema: - -| Type | Fields | Use Case | -|---|---|---| -| `generic` | `key` (required) | General-purpose API key | -| `bearer` | `token` (required) | Bearer token authentication | -| `openai` | `api_key` (required), `org_id` (optional) | OpenAI API access | -| `basic` | `username` (required), `password` (required) | HTTP Basic authentication | - -Types enforce that the right fields are present when you create a credential, reducing misconfiguration errors at runtime. - -### How Credential Resolution Works - -When the engine reaches a step with a `credential` field, it resolves the credential name before invoking the connector: - -1. **Postgres lookup** -- the engine queries the credentials table, decrypts the stored fields using AES-256-GCM, and passes them to the connector. -2. **Environment variable fallback** -- if the credential is not found in Postgres, the engine checks for an environment variable named `MANTLE_SECRET_` (hyphens are replaced with underscores). The env var value is returned as a single `key` field, equivalent to a `generic` credential. - -The resolved credential fields are injected directly into the connector as an internal `_credential` parameter. They are never visible in CEL expressions, step outputs, or execution logs. - -### Security Properties - -- **Encrypted at rest** -- credential field values are encrypted with AES-256-GCM before being written to Postgres. The encryption key is not stored in the database. -- **Never in expressions** -- you cannot reference `credential` data in CEL templates or `if` conditions. The credential is resolved inside the connector, not in the expression engine. -- **Never in logs** -- credential values do not appear in execution logs, step outputs, or error messages. -- **Typed validation** -- creating a credential validates that all required fields for the type are present. - -For the full operational guide, see the [Secrets Guide](secrets-guide.md). - -## Triggers and Server Mode - -Up to this point, every concept on this page describes workflows that are triggered manually: you run `mantle run` and the engine executes the latest applied version. Triggers and server mode introduce automatic execution. - -### Server Mode - -`mantle serve` starts Mantle as a long-running process. Instead of executing a single workflow and exiting, the server stays up and: - -- Accepts HTTP API requests to trigger and cancel executions -- Polls for due cron triggers every 30 seconds -- Listens for incoming webhook requests -- Serves health endpoints for load balancer and Kubernetes probes - -The server runs migrations automatically on startup, so you do not need a separate `mantle init` step in your deployment pipeline. - -### Cron Triggers - -A cron trigger tells Mantle to start a new workflow execution on a recurring schedule. The schedule uses standard 5-field cron syntax (minute, hour, day-of-month, month, day-of-week). - -```yaml -triggers: - - type: cron - schedule: "*/5 * * * *" -``` - -The cron scheduler is built into the `mantle serve` process. It polls every 30 seconds, checks which cron triggers are due, and starts new executions for them. Cron triggers have no effect when running workflows manually with `mantle run` -- they only fire in server mode. - -### Webhook Triggers - -A webhook trigger tells Mantle to start a new workflow execution when an HTTP POST arrives at a specific path. The request body is parsed as JSON and made available as `trigger.payload` in CEL expressions. - -```yaml -triggers: - - type: webhook - path: "/hooks/on-deploy" -``` - -This is the primary way to integrate Mantle with external systems: CI pipelines, monitoring alerts, GitHub webhooks, and third-party SaaS tools can all POST to a webhook endpoint to kick off a workflow. - -### Trigger Lifecycle - -Triggers are managed through the same IaC lifecycle as the rest of the workflow definition. When you run `mantle apply`: - -- Triggers defined in the YAML are registered (or updated) in the database -- Triggers that were previously registered but are no longer in the YAML are deregistered - -This means the workflow YAML file is the single source of truth for trigger configuration. You do not create, update, or delete triggers separately -- they are part of the apply cycle. - -``` -# First apply: registers the cron trigger -mantle apply workflow.yaml - -# Edit workflow.yaml: change schedule from */5 to */10 -mantle apply workflow.yaml # Updates the trigger - -# Edit workflow.yaml: remove the triggers section entirely -mantle apply workflow.yaml # Deregisters all triggers -``` - -### Cron vs Webhook: When to Use Which - -| Use Case | Trigger Type | Why | -|---|---|---| -| Periodic data sync | Cron | Runs on a fixed schedule regardless of external events | -| Deploy notifications | Webhook | Fires in response to an external event (CI pipeline) | -| Daily report generation | Cron | Time-based, no external signal needed | -| GitHub push handler | Webhook | Event-driven, triggered by an external system | -| Scheduled cleanup | Cron | Maintenance task on a recurring schedule | - -Many workflows benefit from both: a cron trigger for periodic runs and a webhook trigger for on-demand execution by external systems. - -## Data Residency - -Mantle is a self-hosted platform. You control where your data lives by choosing where to deploy Postgres and the Mantle binary. - -### Where Data Resides - -All workflow data -- inputs, outputs, step checkpoints, encrypted credentials, and audit events -- is stored in the Postgres database. There is no external data store, no telemetry sent to Anthropic or any third party, and no cloud dependency unless you configure one (e.g., cloud secret backends). - -Data residency is determined entirely by where you host Postgres. Deploy Postgres in the EU, and all Mantle data resides in the EU. - -### BYOK and Credential Storage - -Mantle's Bring Your Own Key (BYOK) model means your API keys and credentials are stored in YOUR database, encrypted with YOUR encryption key. They are not sent to a third-party service for storage or management. This is a fundamental difference from SaaS platforms that hold your credentials on their infrastructure. - -### AI Connector and Cross-Border Data Flow - -While Mantle itself keeps all data in your Postgres instance, the AI connector sends prompts and receives responses from external LLM provider APIs. These API calls cross network boundaries and may cross geographic borders: - -| Provider | Default Endpoint | Data Location | -|---|---|---| -| OpenAI | `https://api.openai.com/v1` | US-based (unless using Azure OpenAI regional endpoints) | -| Azure OpenAI | Configured via `base_url` | Region-specific, controlled by your Azure deployment | -| AWS Bedrock | Configured via `region` param or `aws.region` config | Region-specific (e.g., `eu-west-1`, `us-east-1`) | -| Self-hosted (Ollama, vLLM) | Configured via `base_url` | Wherever you host it | - -### Restricting AI Endpoints - -Use `engine.allowed_base_urls` in your configuration to restrict which AI API endpoints can be called. This prevents workflow authors from accidentally or intentionally sending data to unapproved regions or providers: - -```yaml -# mantle.yaml -engine: - allowed_base_urls: - - "https://bedrock-runtime.eu-west-1.amazonaws.com" - - "https://my-internal-llm.corp.example.com" -``` - -Any `ai/completion` step that specifies a `base_url` not on this list is rejected at validation time. - -### EU Compliance Example - -To keep all data within the EU: - -1. **Deploy Postgres in an EU region** (e.g., AWS `eu-west-1`, GCP `europe-west1`, or an EU-based self-hosted server) -2. **Use EU-region AI endpoints** -- AWS Bedrock in `eu-west-1`, Azure OpenAI in `westeurope`, or a self-hosted model in your EU infrastructure -3. **Restrict endpoints** with `engine.allowed_base_urls` to prevent calls to US-based APIs -4. **Deploy the Mantle binary in the same EU region** to avoid cross-border traffic between the application and the database - -## Architecture Summary - -Mantle is a single Go binary that connects to a Postgres database. Cloud secret stores are optional; plugins run as subprocesses. - -``` -+---------------------------+ +-----------+ -| mantle (binary) |---->| Postgres | -| | | | -| - CLI commands | | - workflow_definitions -| - Workflow engine | | - workflow_executions -| - Built-in connectors | | - step_executions -| - Plugin manager | | - credentials -| - API server + /metrics | | - audit_events -| - Cron scheduler | | - shared_workflows -| - Webhook listener | +-----------+ -| - Audit emitter | -| - Secret resolver |----> Cloud Secret Stores -| | (AWS, GCP, Azure — optional) -+---------------------------+ - | - |--- spawn ---> Plugin subprocesses - (JSON stdin/stdout) -``` - -**Single binary.** No separate worker processes, message queues, or caches. The binary contains the CLI, the execution engine, the connectors, the plugin manager, and the API server. - -**Postgres for everything.** Workflow definitions, execution state, step checkpoints, encrypted credentials, audit events, and shared templates all live in Postgres. This keeps the operational surface area small. - -**Cloud secret stores are optional.** Mantle resolves credentials from Postgres first, then tries configured cloud backends (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault), and finally falls back to environment variables. - -**Plugins are isolated.** Third-party connectors run as subprocesses with a JSON stdin/stdout protocol. They cannot access the engine's memory or database directly. - -## Further Reading - -- [Getting Started](getting-started.md) -- install and run your first workflow -- [Workflow Reference](workflow-reference.md) -- complete YAML schema documentation -- [CLI Reference](cli-reference.md) -- every command and flag -- [Configuration](configuration.md) -- config file, env vars, and flag precedence -- [Secrets Guide](secrets-guide.md) -- credential encryption, cloud backends, and key rotation -- [Server Guide](server-guide.md) -- running Mantle as a persistent server with triggers -- [Plugins Guide](plugins-guide.md) -- writing and managing third-party connector plugins -- [Observability Guide](observability-guide.md) -- Prometheus metrics, audit trail, and structured logging diff --git a/docs/configuration.md b/docs/configuration.md deleted file mode 100644 index 15ef281..0000000 --- a/docs/configuration.md +++ /dev/null @@ -1,264 +0,0 @@ -# Configuration - -Mantle loads configuration from three sources: a YAML config file, environment variables, and CLI flags. This page documents every configuration option and how the sources interact. - -## Precedence - -When the same setting is specified in multiple places, the highest-priority source wins: - -1. **CLI flags** (highest priority) -2. **Environment variables** -3. **Config file** (`mantle.yaml`) -4. **Built-in defaults** (lowest priority) - -For example, if `mantle.yaml` sets `database.url` to one value and you pass `--database-url` on the command line, the flag value is used. - -## Config File - -By default, Mantle looks for a file named `mantle.yaml` in the current working directory. You can specify a different path with the `--config` flag. - -### Full Example - -```yaml -database: - url: postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable - max_open_conns: 25 - max_idle_conns: 10 - conn_max_lifetime: 5m - -api: - address: ":8080" - -log: - level: info - -encryption: - key: "your-64-char-hex-encoded-key-here" - -engine: - worker_poll_interval: 200ms - worker_max_backoff: 5s - orchestrator_poll_interval: 500ms - step_lease_duration: 60s - orchestration_lease_duration: 120s - ai_step_lease_duration: 300s - reaper_interval: 30s - step_output_max_bytes: 1048576 - default_max_tool_rounds: 10 - default_max_tool_calls_per_round: 10 -``` - -### All Config File Fields - -| Field | Type | Default | Description | -|---|---|---|---| -| `database.url` | string | `postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable` | Postgres connection URL. | -| `api.address` | string | `:8080` | Listen address for `mantle serve`. Format: `host:port` or `:port`. Used by the HTTP API, webhook listener, and health endpoints. | -| `log.level` | string | `info` | Log verbosity. One of: `debug`, `info`, `warn`, `error`. | -| `encryption.key` | string | -- | Hex-encoded 32-byte master key for credential encryption. Required for `mantle secrets` commands. Generate with `openssl rand -hex 32`. | -| `database.max_open_conns` | integer | `25` | Maximum number of open connections in the database pool. | -| `database.max_idle_conns` | integer | `10` | Maximum number of idle connections retained in the pool. | -| `database.conn_max_lifetime` | duration | `5m` | Maximum time a connection can be reused before it is closed. Uses Go duration format. | -| `engine.node_id` | string | `hostname:pid` | Unique identifier for this engine node. Auto-generated from hostname and PID if not set. | -| `engine.worker_poll_interval` | duration | `200ms` | How often workers poll for available step work. | -| `engine.worker_max_backoff` | duration | `5s` | Maximum backoff duration for worker polling when no work is available. | -| `engine.orchestrator_poll_interval` | duration | `500ms` | How often the orchestrator polls for workflow executions to advance. | -| `engine.step_lease_duration` | duration | `60s` | How long a worker holds a lease on a step before it can be reclaimed. | -| `engine.orchestration_lease_duration` | duration | `120s` | How long a node holds the orchestration lease for a workflow execution. | -| `engine.ai_step_lease_duration` | duration | `300s` | Lease duration for AI completion steps, which typically take longer than other steps. | -| `engine.reaper_interval` | duration | `30s` | How often the reaper checks for expired leases and stalled executions. | -| `engine.step_output_max_bytes` | integer | `1048576` | Maximum size in bytes for a single step's output. Outputs exceeding this limit are truncated. Default is 1 MB. | -| `engine.default_max_tool_rounds` | integer | `10` | Default maximum number of LLM-tool interaction rounds for AI steps with tools. Can be overridden per step with `max_tool_rounds`. | -| `engine.default_max_tool_calls_per_round` | integer | `10` | Default maximum number of tool calls the LLM can make per round. Can be overridden per step with `max_tool_calls_per_round`. | - -### Config File Discovery - -When you do not pass `--config`, Mantle searches for `mantle.yaml` in the current directory. If no config file is found, Mantle silently falls back to defaults. This is intentional -- most commands work fine with defaults when you use the provided `docker-compose.yml`. - -When you pass `--config path/to/config.yaml` explicitly, Mantle requires that file to exist and be valid YAML. A missing or unparseable explicit config file is a hard error. - -## Environment Variables - -All environment variables use the `MANTLE_` prefix with underscores replacing dots and hyphens. - -| Env Var | Config File Equivalent | Default | -|---|---|---| -| `MANTLE_DATABASE_URL` | `database.url` | `postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable` | -| `MANTLE_API_ADDRESS` | `api.address` | `:8080` | -| `MANTLE_LOG_LEVEL` | `log.level` | `info` | -| `MANTLE_ENCRYPTION_KEY` | `encryption.key` | -- | -| `MANTLE_DATABASE_MAX_OPEN_CONNS` | `database.max_open_conns` | `25` | -| `MANTLE_DATABASE_MAX_IDLE_CONNS` | `database.max_idle_conns` | `10` | -| `MANTLE_DATABASE_CONN_MAX_LIFETIME` | `database.conn_max_lifetime` | `5m` | -| `MANTLE_ENGINE_NODE_ID` | `engine.node_id` | `hostname:pid` | -| `MANTLE_ENGINE_WORKER_POLL_INTERVAL` | `engine.worker_poll_interval` | `200ms` | -| `MANTLE_ENGINE_WORKER_MAX_BACKOFF` | `engine.worker_max_backoff` | `5s` | -| `MANTLE_ENGINE_ORCHESTRATOR_POLL_INTERVAL` | `engine.orchestrator_poll_interval` | `500ms` | -| `MANTLE_ENGINE_STEP_LEASE_DURATION` | `engine.step_lease_duration` | `60s` | -| `MANTLE_ENGINE_ORCHESTRATION_LEASE_DURATION` | `engine.orchestration_lease_duration` | `120s` | -| `MANTLE_ENGINE_AI_STEP_LEASE_DURATION` | `engine.ai_step_lease_duration` | `300s` | -| `MANTLE_ENGINE_REAPER_INTERVAL` | `engine.reaper_interval` | `30s` | -| `MANTLE_ENGINE_STEP_OUTPUT_MAX_BYTES` | `engine.step_output_max_bytes` | `1048576` | -| `MANTLE_ENGINE_DEFAULT_MAX_TOOL_ROUNDS` | `engine.default_max_tool_rounds` | `10` | -| `MANTLE_ENGINE_DEFAULT_MAX_TOOL_CALLS_PER_ROUND` | `engine.default_max_tool_calls_per_round` | `10` | - -**Example:** - -```bash -export MANTLE_DATABASE_URL="postgres://prod:secret@db.example.com:5432/mantle?sslmode=require" -export MANTLE_LOG_LEVEL="warn" -mantle init -``` - -Environment variables are useful in container deployments and CI pipelines where you do not want to mount a config file. - -## CLI Flags - -Every configuration option has a corresponding CLI flag. Flags take the highest priority. - -| Flag | Config File Equivalent | Default | -|---|---|---| -| `--database-url` | `database.url` | `postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable` | -| `--api-address` | `api.address` | `:8080` | -| `--log-level` | `log.level` | `info` | -| `--config` | -- | `mantle.yaml` (current directory) | - -The `--config` flag has no environment variable or config file equivalent -- it controls which config file to load. - -**Example:** - -```bash -mantle init --database-url "postgres://prod:secret@db.example.com:5432/mantle?sslmode=require" -``` - -## Defaults - -If you start Postgres using the included `docker-compose.yml` and run Mantle from the project directory, the defaults work without any configuration: - -| Setting | Default Value | -|---|---| -| Database URL | `postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable` | -| API address | `:8080` | -| Log level | `info` | - -## Common Configurations - -### Local Development - -No config file needed. Start Postgres with `docker-compose up -d` and use defaults: - -```bash -mantle init -mantle validate workflow.yaml -mantle apply workflow.yaml -``` - -### CI Pipeline - -Use environment variables to avoid config files in CI: - -```bash -export MANTLE_DATABASE_URL="postgres://ci:ci@localhost:5432/mantle_test?sslmode=disable" -mantle init -mantle validate workflow.yaml -mantle apply workflow.yaml -``` - -### Production - -Use a `mantle.yaml` file with production values, or pass everything through environment variables: - -```yaml -# mantle.yaml -database: - url: postgres://mantle:${DB_PASSWORD}@db.internal:5432/mantle?sslmode=require - -api: - address: ":8080" - -log: - level: warn -``` - -Note: Mantle does not perform variable substitution in the config file. The `${DB_PASSWORD}` example above is illustrative -- use environment variables (`MANTLE_DATABASE_URL`) for secrets instead of embedding them in config files. - -For production secrets management, set the encryption key through an environment variable rather than the config file: - -```bash -export MANTLE_ENCRYPTION_KEY="$(openssl rand -hex 32)" -export MANTLE_DATABASE_URL="postgres://mantle:secret@db.internal:5432/mantle?sslmode=require" -mantle secrets create --name prod-openai --type openai --field api_key=sk-... -``` - -The encryption key has no default value. It is only required when you use `mantle secrets` commands or run workflows that reference credentials. All other Mantle commands work without it. - -### Server Mode - -When running `mantle serve`, the `api.address` setting controls which address the HTTP server, webhook listener, and health endpoints bind to: - -```yaml -# mantle.yaml -database: - url: postgres://mantle:secret@db.internal:5432/mantle?sslmode=require - -api: - address: ":8080" - -log: - level: info -``` - -Or with environment variables: - -```bash -export MANTLE_DATABASE_URL="postgres://mantle:secret@db.internal:5432/mantle?sslmode=require" -export MANTLE_API_ADDRESS=":8080" -export MANTLE_ENCRYPTION_KEY="$(cat /run/secrets/mantle-key)" -mantle serve -``` - -The server runs migrations automatically on startup, so you do not need a separate `mantle init` step. - -### Offline Commands - -The `mantle validate` and `mantle version` commands do not require a database connection. They skip config loading entirely, so you can run them without Postgres, a config file, or any environment variables: - -```bash -mantle validate workflow.yaml # works anywhere, no database needed -mantle version # works anywhere -``` - -## Cloud Secret Backend Configuration - -Mantle can resolve credentials from external cloud secret stores. Cloud backends are configured through environment variables -- there are no config file fields for these. - -| Env Var | Description | -|---|---| -| `AWS_REGION` or `AWS_DEFAULT_REGION` | AWS region for Secrets Manager. Enables the AWS backend when AWS credentials are available. | -| `MANTLE_GCP_PROJECT` | GCP project ID for Secret Manager. Enables the GCP backend. | -| `MANTLE_AZURE_VAULT_URL` | Azure Key Vault URL (e.g., `https://my-vault.vault.azure.net/`). Enables the Azure backend. | - -Cloud backends are optional. If none are configured, Mantle resolves credentials from the Postgres store and environment variable fallback only. - -Each cloud backend uses its respective SDK's default credential chain: - -- **AWS**: environment variables, shared credentials file, EC2/ECS instance profile -- **GCP**: Application Default Credentials (`gcloud auth application-default login`, Workload Identity, service account key) -- **Azure**: DefaultAzureCredential (`az login`, managed identity, environment variables) - -See the [Secrets Guide](secrets-guide.md#cloud-secret-backends) for detailed setup instructions and IAM requirements. - -## Plugin Configuration - -Plugins are stored in `.mantle/plugins/` relative to the current working directory. This location is not currently configurable via the config file -- it uses a fixed path. - -## Reference - -See also: - -- [CLI Reference](cli-reference.md) -- flag documentation for every command -- [Getting Started](getting-started.md) -- setup walkthrough using defaults -- [Secrets Guide](secrets-guide.md) -- credential encryption, cloud backends, and key rotation -- [Server Guide](server-guide.md) -- running Mantle as a persistent server with triggers -- [Plugins Guide](plugins-guide.md) -- writing and managing third-party connector plugins -- [Observability Guide](observability-guide.md) -- Prometheus metrics, audit trail, and structured logging diff --git a/docs/deployment-guide.md b/docs/deployment-guide.md deleted file mode 100644 index f02e03e..0000000 --- a/docs/deployment-guide.md +++ /dev/null @@ -1,315 +0,0 @@ -# Deployment Guide - -This guide covers installing Mantle, running it in production, and hardening your deployment. For trigger configuration and the REST API, see the [Server Guide](server-guide.md). For authentication setup, see the [Authentication & RBAC Guide](authentication-guide.md). - -## Prerequisites - -Before deploying Mantle, you need: - -- **Postgres 14+** -- Mantle's single point of state. All workflow definitions, executions, credentials, and audit events live in the database. -- **Encryption key** -- a 32-byte hex-encoded key for encrypting credentials at rest. See the [Secrets Guide](secrets-guide.md) for details. -- **Domain and TLS** -- required for production. Mantle can terminate TLS directly or run behind a reverse proxy. - -## Installation Methods - -### Binary Download - -Download the latest release from [GitHub Releases](https://github.com/dvflw/mantle/releases): - -```bash -# Linux (amd64) -curl -Lo mantle https://github.com/dvflw/mantle/releases/latest/download/mantle-linux-amd64 -chmod +x mantle -sudo mv mantle /usr/local/bin/ - -# macOS (Apple Silicon) -curl -Lo mantle https://github.com/dvflw/mantle/releases/latest/download/mantle-darwin-arm64 -chmod +x mantle -sudo mv mantle /usr/local/bin/ -``` - -Verify the installation: - -```bash -mantle version -# mantle v0.1.0 (791fa83, built 2026-03-18T00:00:00Z) -``` - -### Go Install - -If you have Go 1.25+ installed: - -```bash -go install github.com/dvflw/mantle/cmd/mantle@latest -``` - -The binary is placed in `$GOPATH/bin` (or `$HOME/go/bin` by default). - -### Docker - -Pull the official image: - -```bash -docker pull ghcr.io/dvflw/mantle:0.1.0 -``` - -Run with environment variables: - -```bash -docker run -d \ - -p 8080:8080 \ - -e MANTLE_DATABASE_URL="postgres://mantle:secret@host.docker.internal:5432/mantle?sslmode=disable" \ - -e MANTLE_ENCRYPTION_KEY="your-64-char-hex-key" \ - ghcr.io/dvflw/mantle:0.1.0 serve -``` - -### Helm Chart - -For Kubernetes deployments, use the included Helm chart. See [Production Deployment (Kubernetes/Helm)](#production-deployment-kuberneteshelm) below. - -## Quick Start (Docker Compose) - -The fastest way to get Mantle running locally with Postgres: - -```bash -git clone https://github.com/dvflw/mantle.git && cd mantle -docker compose up -d -``` - -This starts Postgres 16 on `localhost:5432` with user `mantle`, password `mantle`, and database `mantle`. - -Run migrations and start the server: - -```bash -export MANTLE_DATABASE_URL="postgres://mantle:mantle@localhost:5432/mantle?sslmode=disable" -export MANTLE_ENCRYPTION_KEY=$(openssl rand -hex 32) - -mantle init -mantle serve -``` - -``` -Running migrations... -Migrations complete. -Starting server on :8080 -Cron scheduler started (poll interval: 30s) -``` - -Mantle is now running at `http://localhost:8080`. See the [Getting Started](getting-started.md) guide for your first workflow. - -## Production Deployment (Kubernetes/Helm) - -The recommended way to run Mantle in production is with the included Helm chart. - -### Basic Install - -```bash -helm install mantle charts/mantle \ - --set database.url="postgres://mantle:secret@db.internal:5432/mantle?sslmode=require" \ - --set encryption.key="your-64-char-hex-key" \ - --set replicaCount=3 -``` - -### Values Reference - -| Value | Default | Description | -|---|---|---| -| `image.repository` | `ghcr.io/dvflw/mantle` | Container image repository | -| `image.tag` | Chart `appVersion` | Container image tag | -| `image.pullPolicy` | `IfNotPresent` | Image pull policy | -| `replicaCount` | `1` | Number of server replicas | -| `database.url` | -- | Postgres connection string (required) | -| `encryption.key` | -- | 32-byte hex encryption key (required) | -| `resources.requests.cpu` | `100m` | CPU request | -| `resources.requests.memory` | `128Mi` | Memory request | -| `resources.limits.cpu` | `500m` | CPU limit | -| `resources.limits.memory` | `512Mi` | Memory limit | -| `probes.liveness.path` | `/healthz` | Liveness probe path | -| `probes.readiness.path` | `/readyz` | Readiness probe path | -| `pdb.enabled` | `false` | Enable PodDisruptionBudget | -| `pdb.minAvailable` | `1` | Minimum available pods during disruption | -| `securityContext.runAsNonRoot` | `true` | Run as non-root user | -| `securityContext.readOnlyRootFilesystem` | `true` | Read-only root filesystem | -| `ingress.enabled` | `false` | Enable Ingress resource | -| `ingress.className` | `""` | Ingress class name | -| `ingress.hosts` | `[]` | Ingress host rules | -| `ingress.tls` | `[]` | Ingress TLS configuration | - -### TLS Termination - -**Option 1: Ingress (recommended for Kubernetes).** Configure TLS at the Ingress level: - -```yaml -# values.yaml -ingress: - enabled: true - className: nginx - hosts: - - host: mantle.company.com - paths: - - path: / - pathType: Prefix - tls: - - secretName: mantle-tls - hosts: - - mantle.company.com -``` - -**Option 2: Native TLS.** Mantle can terminate TLS directly: - -```bash -mantle serve --tls-cert /etc/mantle/tls.crt --tls-key /etc/mantle/tls.key -``` - -Or via environment variables: - -```bash -export MANTLE_TLS_CERT="/etc/mantle/tls.crt" -export MANTLE_TLS_KEY="/etc/mantle/tls.key" -``` - -### Database Migrations - -The Helm chart includes a pre-install/pre-upgrade hook Job that runs `mantle init` before the new version starts serving traffic. You do not need to run migrations separately. The migration job uses database-level locking, so it is safe with multiple replicas. - -## Configuration Checklist - -Use this checklist when deploying to production. Every item maps to an environment variable or `mantle.yaml` field. - -| Setting | Env Var | Required | Notes | -|---|---|---|---| -| Database URL | `MANTLE_DATABASE_URL` | Yes | Use `sslmode=require` in production | -| Encryption key | `MANTLE_ENCRYPTION_KEY` | Yes | 32 bytes, hex-encoded (64 characters) | -| API listen address | `MANTLE_API_ADDRESS` | No | Default `:8080` | -| TLS certificate | `MANTLE_TLS_CERT` | No | Path to PEM-encoded certificate | -| TLS private key | `MANTLE_TLS_KEY` | No | Path to PEM-encoded private key | -| OIDC issuer URL | `MANTLE_AUTH_OIDC_ISSUER_URL` | No | Required if using SSO | -| OIDC client ID | `MANTLE_AUTH_OIDC_CLIENT_ID` | No | Required if using SSO | -| OIDC audience | `MANTLE_AUTH_OIDC_AUDIENCE` | No | Required if using SSO | -| OIDC allowed domains | `MANTLE_AUTH_OIDC_ALLOWED_DOMAINS` | No | Comma-separated domain list | -| AWS region | `AWS_REGION` | No | Required if using AWS Secrets Manager | -| GCP project | `MANTLE_GCP_PROJECT` | No | Required if using GCP Secret Manager | -| Azure vault URL | `MANTLE_AZURE_VAULT_URL` | No | Required if using Azure Key Vault | -| Execution retention | `MANTLE_RETENTION_EXECUTION_DAYS` | No | Default `90`. Days to keep completed executions | -| Audit retention | `MANTLE_RETENTION_AUDIT_DAYS` | No | Default `365`. Days to keep audit events | -| Allowed AI base URLs | `MANTLE_AI_ALLOWED_BASE_URLS` | No | Comma-separated. Restricts which AI endpoints connectors can call | -| Log level | `MANTLE_LOG_LEVEL` | No | Default `info`. Options: `debug`, `info`, `warn`, `error` | - -## Production Hardening - -### Enable TLS - -Never run Mantle over plain HTTP in production. Use one of: - -- Ingress with TLS termination (see [TLS Termination](#tls-termination) above) -- Native TLS with `--tls-cert` and `--tls-key` -- A reverse proxy (nginx, Caddy, HAProxy) that terminates TLS - -### Set Resource Limits - -Always set CPU and memory limits to prevent a single workflow execution from consuming all available resources: - -```yaml -# values.yaml -resources: - requests: - cpu: 250m - memory: 256Mi - limits: - cpu: "1" - memory: 1Gi -``` - -### Configure PodDisruptionBudget - -For high availability, enable the PDB to ensure at least one replica stays available during node maintenance: - -```yaml -# values.yaml -replicaCount: 3 -pdb: - enabled: true - minAvailable: 1 -``` - -### Set Retention Policies - -Configure retention to prevent unbounded storage growth: - -```yaml -# mantle.yaml -retention: - execution_days: 90 - audit_days: 365 -``` - -Or via environment variables: - -```bash -export MANTLE_RETENTION_EXECUTION_DAYS=90 -export MANTLE_RETENTION_AUDIT_DAYS=365 -``` - -Mantle runs a background cleanup job that deletes completed executions and audit events older than the configured retention period. - -### Restrict AI Base URLs - -If your workflows use the AI connector, restrict which endpoints can be called to prevent exfiltration: - -```yaml -# mantle.yaml -ai: - allowed_base_urls: - - "https://api.openai.com" - - "https://api.anthropic.com" -``` - -Requests to any other base URL are rejected at execution time. - -### Monitor with Prometheus - -Scrape the `/metrics` endpoint with Prometheus or a compatible collector: - -```yaml -# prometheus.yml -scrape_configs: - - job_name: mantle - static_configs: - - targets: ["mantle:8080"] -``` - -See the [Observability Guide](observability-guide.md) for metric names, example PromQL queries, and Grafana dashboard configuration. - -### Set Up Database Backups - -Postgres is Mantle's single point of state. Set up automated backups: - -- **Managed Postgres (RDS, Cloud SQL, Azure Database):** Enable automated snapshots and WAL archiving -- **Self-hosted Postgres:** Schedule `pg_dump` and configure WAL archiving - -See the [Server Guide](server-guide.md#backup-and-disaster-recovery) for detailed backup procedures, recovery steps, and RPO/RTO guidance. - -### Security Context - -The Helm chart defaults to a secure pod configuration: - -```yaml -securityContext: - runAsNonRoot: true - readOnlyRootFilesystem: true - allowPrivilegeEscalation: false - capabilities: - drop: - - ALL -``` - -Do not override these defaults unless you have a specific requirement. - -## Further Reading - -- [Getting Started](getting-started.md) -- first workflow in under five minutes -- [Server Guide](server-guide.md) -- triggers, REST API, graceful shutdown, backup and recovery -- [Authentication & RBAC Guide](authentication-guide.md) -- API keys, OIDC/SSO, roles, and teams -- [Secrets Guide](secrets-guide.md) -- credential encryption, cloud backends, key rotation -- [Configuration](configuration.md) -- full configuration reference -- [Observability Guide](observability-guide.md) -- Prometheus metrics, audit trail, structured logging diff --git a/docs/getting-started.md b/docs/getting-started.md deleted file mode 100644 index 2b1bfea..0000000 --- a/docs/getting-started.md +++ /dev/null @@ -1,539 +0,0 @@ -# Getting Started - -After completing this guide, you will have Mantle installed, a Postgres database running, and your first workflow executing -- all in under five minutes. From there, the guide progressively introduces data passing, conditional logic, AI/LLM integration, server mode, and multi-tenancy. - -## What is Mantle? - -Mantle is a headless AI workflow automation platform. You define workflows as YAML, deploy them through an infrastructure-as-code lifecycle (validate, plan, apply), and execute them against a Postgres-backed engine. It ships as a single Go binary -- bring your own API keys, bring your own database, no hosted runtime required. - -## Prerequisites - -You need the following installed on your machine: - -- **Go 1.25+** -- [install instructions](https://go.dev/doc/install) -- **Docker and Docker Compose** -- [install instructions](https://docs.docker.com/get-docker/) -- **Make** -- included on macOS and most Linux distributions - -Verify your setup: - -```bash -go version # go1.25 or later -docker --version -``` - -## Install and Start (< 2 minutes) - -Clone the repository, start Postgres, build the binary, and run migrations: - -```bash -git clone https://github.com/dvflw/mantle.git && cd mantle -docker compose up -d -make build -./mantle init -``` - -The `docker compose up -d` command starts Postgres 16 on `localhost:5432` with user `mantle`, password `mantle`, and database `mantle`. The `make build` command produces a single `mantle` binary in the project root. The `mantle init` command creates all required database tables. - -You should see: - -``` -Running migrations... -Migrations complete. -``` - -Optionally, move the binary onto your PATH: - -```bash -sudo mv mantle /usr/local/bin/ -``` - -Verify it works: - -```bash -mantle version -# mantle v0.1.0 (791fa83, built 2026-03-18T00:00:00Z) -``` - -## Your First Workflow (< 3 minutes) - -The `examples/` directory includes several ready-to-run workflows. Start with the simplest one -- a single HTTP GET request. - -Look at `examples/hello-world.yaml`: - -```yaml -name: hello-world -description: Fetch a random fact from a public API — the simplest possible Mantle workflow - -steps: - - name: fetch - action: http/request - params: - method: GET - url: "https://jsonplaceholder.typicode.com/posts/1" -``` - -This workflow has one step: it sends a GET request to the JSONPlaceholder API and returns the response. - -### Step 1: Validate - -Check the workflow for structural errors. This runs offline -- no database connection required: - -```bash -mantle validate examples/hello-world.yaml -``` - -Output: - -``` -hello-world.yaml: valid -``` - -If there are errors, Mantle reports them with file, line, and column numbers: - -``` -bad-workflow.yaml:1:1: error: name must match ^[a-z][a-z0-9-]*$ (name) -``` - -### Step 2: Apply - -Store the workflow definition as a new immutable version in the database: - -```bash -mantle apply examples/hello-world.yaml -``` - -Output: - -``` -Applied hello-world version 1 -``` - -Every time you edit a workflow and re-apply, Mantle creates a new version. If the content has not changed, it tells you: - -``` -No changes — hello-world is already at version 1 -``` - -You can also preview what will change before applying: - -```bash -mantle plan examples/hello-world.yaml -``` - -``` -No changes — hello-world is at version 1 -``` - -### Step 3: Run - -Execute the workflow by name: - -```bash -mantle run hello-world -``` - -Output: - -``` -Running hello-world (version 1)... -Execution a1b2c3d4-e5f6-7890-abcd-ef1234567890: completed - fetch: completed -``` - -### Step 4: View Logs - -Inspect the execution with the execution ID from the previous step: - -```bash -mantle logs a1b2c3d4-e5f6-7890-abcd-ef1234567890 -``` - -Output: - -``` -Execution: a1b2c3d4-e5f6-7890-abcd-ef1234567890 -Workflow: hello-world (version 1) -Status: completed -Started: 2026-03-18T14:30:00Z -Completed: 2026-03-18T14:30:01Z -Duration: 1.042s - -Steps: - fetch completed (1.0s) -``` - -If a step fails, the error appears below the step name: - -``` -Steps: - fetch failed (0.5s) - error: http/request: GET https://jsonplaceholder.typicode.com/posts/1: connection refused -``` - -You can also get a quick status summary with `mantle status `. - -## Data Passing Between Steps - -Workflows become powerful when steps pass data to each other. Look at `examples/chained-requests.yaml`: - -```yaml -name: chained-requests -description: > - Fetch a user from a public API, then fetch their posts using the user's ID. - Demonstrates CEL data passing between steps via steps..output. - -steps: - - name: get-user - action: http/request - params: - method: GET - url: "https://jsonplaceholder.typicode.com/users/1" - - - name: get-user-posts - action: http/request - params: - method: GET - url: "https://jsonplaceholder.typicode.com/posts?userId={{ steps['get-user'].output.json.id }}" -``` - -The key line is the second step's URL. The expression `{{ steps['get-user'].output.json.id }}` reads the JSON response from the `get-user` step and extracts the `id` field. - -Apply and run it: - -```bash -mantle apply examples/chained-requests.yaml -mantle run chained-requests -``` - -``` -Running chained-requests (version 1)... -Execution b2c3d4e5-f6a7-8901-bcde-f12345678901: completed - get-user: completed - get-user-posts: completed -``` - -### CEL Expression Syntax - -Mantle uses [CEL (Common Expression Language)](https://github.com/google/cel-go) for data passing and conditional logic. The essentials: - -- **Access step output:** `steps['step-name'].output.json.field` -- **Access inputs:** `inputs.field_name` -- **Bracket notation is required** when step names contain hyphens: `steps['get-user']` (not `steps.get-user`) -- **Dot notation works** for step names without hyphens: `steps.summarize.output.json.summary` -- **Template strings** use `{{ }}` delimiters inside `params` values - -## Conditional Execution - -Steps can run conditionally based on the output of previous steps. Look at `examples/conditional-workflow.yaml`: - -```yaml -name: conditional-workflow -description: > - Fetch todos for a user, then conditionally post a summary only if there are - incomplete todos. Demonstrates conditional execution with if: and retry policies. - -inputs: - user_id: - type: string - description: JSONPlaceholder user ID (1-10) - -steps: - - name: get-todos - action: http/request - timeout: "10s" - retry: - max_attempts: 3 - backoff: exponential - params: - method: GET - url: "https://jsonplaceholder.typicode.com/todos?userId={{ inputs.user_id }}" - - - name: post-summary - action: http/request - if: "steps['get-todos'].output.status == 200" - params: - method: POST - url: "https://jsonplaceholder.typicode.com/posts" - headers: - Content-Type: "application/json" - body: - title: "Todo summary" - body: "Fetched todos for user {{ inputs.user_id }}" - userId: "{{ inputs.user_id }}" -``` - -This workflow introduces three features: - -- **`inputs`** -- the workflow declares a `user_id` input, passed at runtime with `--input` -- **`if`** -- the `post-summary` step only runs when the CEL expression evaluates to true -- **`retry` and `timeout`** -- the `get-todos` step retries up to 3 times with exponential backoff and times out after 10 seconds - -Apply and run it: - -```bash -mantle apply examples/conditional-workflow.yaml -mantle run conditional-workflow --input user_id=3 -``` - -``` -Running conditional-workflow (version 1)... -Execution c3d4e5f6-a7b8-9012-cdef-123456789012: completed - get-todos: completed - post-summary: completed -``` - -You can pass multiple inputs by repeating the `--input` flag: - -```bash -mantle run my-workflow --input key1=value1 --input key2=value2 -``` - -## Using AI/LLM - -Mantle includes a built-in AI connector that supports OpenAI-compatible APIs. Before using it, you need to store your API key as an encrypted credential. - -### Set Up Credentials - -Generate an encryption key and export it: - -```bash -export MANTLE_ENCRYPTION_KEY=$(openssl rand -hex 32) -``` - -Store your OpenAI API key: - -```bash -mantle secrets create --name openai --type openai --field api_key=sk-proj-your-key-here -``` - -``` -Created credential "openai" (type: openai) -``` - -The credential is encrypted at rest with AES-256-GCM. The raw API key is never stored in plaintext, never exposed in logs, and never available in CEL expressions. See the [Secrets Guide](secrets-guide.md) for credential types and the full security model. - -### AI Completion Step - -Here is a workflow that fetches a webpage and uses an LLM to extract structured data (from `examples/ai-structured-extraction.yaml`): - -```yaml -name: ai-structured-extraction -description: > - Fetch a webpage and use an LLM with output_schema to extract structured data - (title, author, key topics). Demonstrates enforcing JSON structure from AI output. - -inputs: - url: - type: string - description: URL of the page to fetch and extract data from - -steps: - - name: fetch-page - action: http/request - timeout: "15s" - retry: - max_attempts: 2 - backoff: exponential - params: - method: GET - url: "{{ inputs.url }}" - - - name: extract-metadata - action: ai/completion - credential: openai - params: - model: gpt-4o - system_prompt: > - You are a structured data extraction engine. Given raw page content, - extract the requested fields accurately. If a field cannot be determined, - use null or an empty value as appropriate. - prompt: > - Extract the following metadata from this page content: - - {{ steps['fetch-page'].output.body }} - output_schema: - type: object - properties: - title: - type: string - author: - type: string - key_topics: - type: array - items: - type: string - required: - - title - - author - - key_topics - additionalProperties: false -``` - -The `credential: openai` field tells the engine to resolve the `openai` credential you created earlier. The `output_schema` field enforces structured JSON output from the model -- the response is guaranteed to match the schema. - -Apply and run it: - -```bash -mantle apply examples/ai-structured-extraction.yaml -mantle run ai-structured-extraction --input url=https://example.com -``` - -``` -Running ai-structured-extraction (version 1)... -Execution d4e5f6a7-b8c9-0123-defa-234567890123: completed - fetch-page: completed - extract-metadata: completed -``` - -### Key AI Connector Details - -| Field | Description | -|---|---| -| `action` | `ai/completion` for chat completions | -| `credential` | Name of a stored credential (type `openai`) | -| `model` | Model name (e.g., `gpt-4o`, `gpt-4o-mini`) | -| `prompt` | User message -- supports CEL template expressions | -| `system_prompt` | Optional system message to set model behavior | -| `output_schema` | Optional JSON Schema -- enforces structured output | - -## Server Mode and Triggers - -So far you have been running workflows manually with `mantle run`. In production, you start Mantle as a persistent server that supports cron schedules and webhook triggers. - -### Define Triggers - -Add a `triggers` section to your workflow YAML: - -```yaml -name: api-health-check -description: Check API health hourly and on demand - -triggers: - - type: cron - schedule: "0 * * * *" - - type: webhook - path: "/hooks/api-health-check" - -steps: - - name: check-api - action: http/request - timeout: "10s" - retry: - max_attempts: 3 - backoff: exponential - params: - method: GET - url: https://api.example.com/health - - - name: alert-on-failure - action: http/request - if: "steps['check-api'].output.status != 200" - params: - method: POST - url: https://hooks.slack.com/services/T00/B00/xxx - body: - text: "API health check failed with status {{ steps['check-api'].output.status }}" -``` - -Apply the workflow, then start the server: - -```bash -mantle apply api-health-check.yaml -mantle serve -``` - -``` -Running migrations... -Migrations complete. -Starting server on :8080 -Cron scheduler started (poll interval: 30s) -``` - -The server runs migrations on startup, starts the HTTP API on `:8080`, and polls for due cron triggers every 30 seconds. The `api-health-check` workflow now runs every hour automatically. - -### Trigger a Webhook - -Send a POST request to the webhook path: - -```bash -curl -X POST http://localhost:8080/hooks/api-health-check \ - -H "Content-Type: application/json" \ - -d '{"reason": "manual check"}' -``` - -The request body is available as `trigger.payload` in CEL expressions within the workflow. - -### REST API - -The server also exposes a REST API for programmatic access: - -```bash -# Trigger a workflow -curl -s -X POST http://localhost:8080/api/v1/run/api-health-check | jq . -``` - -```json -{ - "execution_id": "e5f6a7b8-c9d0-1234-efab-345678901234", - "workflow": "api-health-check", - "version": 1 -} -``` - -```bash -# Cancel a running execution -curl -s -X POST http://localhost:8080/api/v1/cancel/e5f6a7b8-c9d0-1234-efab-345678901234 -``` - -Health endpoints are available at `/healthz` (liveness) and `/readyz` (readiness, checks database connectivity). See the [Server Guide](server-guide.md) for production deployment, Helm chart configuration, and graceful shutdown behavior. - -## Multi-Tenancy - -Mantle supports teams, users, roles, and API keys for multi-tenant environments. - -Create a team, add a user, and generate an API key: - -```bash -mantle teams create --name acme-corp -``` - -``` -Created team acme-corp (id: f6a7b8c9-d0e1-2345-fabc-456789012345) -``` - -```bash -mantle users create --email alice@acme.com --name "Alice Chen" --team acme-corp --role admin -``` - -``` -Created user alice@acme.com (role: admin, team: acme-corp) -``` - -```bash -mantle users api-key --email alice@acme.com --key-name production -``` - -``` -API Key: mk_a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2 - -Save this key — it cannot be retrieved again. -Key prefix for reference: mk_a1b2c3 -``` - -Available roles are `admin`, `team_owner`, and `operator`. API keys use the `mk_` prefix and are hashed before storage -- the raw key is only shown once at creation time. - -This is a brief overview. Multi-tenancy, role-based access control, and team scoping are covered in detail in the [CLI Reference](cli-reference.md). - -## Next Steps - -You have gone from zero to running workflows with data passing, conditional logic, AI integration, server mode, and multi-tenancy. Here is where to go next: - -- **[Workflow Reference](workflow-reference.md)** -- complete YAML schema: every field, every validation rule, every connector (HTTP, AI, Slack, Postgres, Email, S3) -- **[CLI Reference](cli-reference.md)** -- every command, flag, and the REST API -- **[Secrets Guide](secrets-guide.md)** -- credential types, encryption setup, cloud backends (AWS, GCP, Azure), and key rotation -- **[Server Guide](server-guide.md)** -- production deployment, Helm chart, cron and webhook triggers, REST API -- **[Concepts](concepts.md)** -- architecture, checkpointing, CEL expressions, versioning, connectors, plugins, and observability -- **[Plugins Guide](plugins-guide.md)** -- extend Mantle with third-party connector plugins -- **[Observability Guide](observability-guide.md)** -- Prometheus metrics, audit trail, and structured logging -- **[Configuration](configuration.md)** -- config file, environment variables, cloud backends, and flag precedence -- **[examples/](https://github.com/dvflw/mantle/tree/main/examples)** -- ready-to-run workflow files covering HTTP, AI, chained requests, and more diff --git a/docs/observability-guide.md b/docs/observability-guide.md deleted file mode 100644 index 30ca222..0000000 --- a/docs/observability-guide.md +++ /dev/null @@ -1,354 +0,0 @@ -# Observability Guide - -Mantle provides three observability mechanisms: Prometheus metrics, an immutable audit trail, and structured JSON logging. This guide covers how to set up each one, what data is available, and how to integrate with external monitoring systems. - -## Prometheus Metrics - -When running in server mode (`mantle serve`), Mantle exposes a `/metrics` endpoint in Prometheus exposition format. The endpoint is available on the same address as the API server (default `:8080`). - -### Scraping Metrics - -Add Mantle to your Prometheus scrape configuration: - -```yaml -# prometheus.yml -scrape_configs: - - job_name: 'mantle' - scrape_interval: 15s - static_configs: - - targets: ['localhost:8080'] -``` - -Verify the endpoint is working: - -```bash -curl http://localhost:8080/metrics -``` - -### Available Metrics - -| Metric | Type | Labels | Description | -|---|---|---|---| -| `mantle_workflow_executions_total` | Counter | `workflow`, `status` | Total number of workflow executions, labeled by workflow name and final status (`completed`, `failed`, `cancelled`). | -| `mantle_step_executions_total` | Counter | `workflow`, `step`, `status` | Total number of step executions, labeled by workflow name, step name, and final status. | -| `mantle_step_duration_seconds` | Histogram | `workflow`, `step`, `action` | Duration of step executions in seconds. Uses default Prometheus histogram buckets. | -| `mantle_connector_duration_seconds` | Histogram | `action` | Duration of connector invocations in seconds, labeled by action name (e.g., `http/request`, `ai/completion`). | -| `mantle_active_executions` | Gauge | -- | Number of currently running workflow executions. | - -### Example PromQL Queries - -**Workflow execution rate (per minute):** - -```promql -rate(mantle_workflow_executions_total[5m]) * 60 -``` - -**Failure rate for a specific workflow:** - -```promql -rate(mantle_workflow_executions_total{workflow="daily-report", status="failed"}[1h]) -/ -rate(mantle_workflow_executions_total{workflow="daily-report"}[1h]) -``` - -**P95 step duration for a workflow:** - -```promql -histogram_quantile(0.95, rate(mantle_step_duration_seconds_bucket{workflow="daily-report"}[5m])) -``` - -**Average connector latency by action type:** - -```promql -rate(mantle_connector_duration_seconds_sum[5m]) -/ -rate(mantle_connector_duration_seconds_count[5m]) -``` - -**Currently active executions:** - -```promql -mantle_active_executions -``` - -**Slowest connectors (P99):** - -```promql -histogram_quantile(0.99, sum by (action, le) (rate(mantle_connector_duration_seconds_bucket[5m]))) -``` - -### Grafana Dashboard - -A useful Grafana dashboard for Mantle includes these panels: - -1. **Execution rate** -- `rate(mantle_workflow_executions_total[5m])` stacked by workflow -2. **Failure rate** -- `rate(mantle_workflow_executions_total{status="failed"}[5m])` with alert threshold -3. **Active executions** -- `mantle_active_executions` as a stat panel -4. **Step duration heatmap** -- `mantle_step_duration_seconds_bucket` as a heatmap -5. **Connector latency** -- `mantle_connector_duration_seconds` by action as a time series - -### Alerting Examples - -**Alert: workflow failure rate exceeds 10%:** - -```yaml -# Prometheus alerting rule -groups: - - name: mantle - rules: - - alert: MantleHighFailureRate - expr: | - rate(mantle_workflow_executions_total{status="failed"}[15m]) - / - rate(mantle_workflow_executions_total[15m]) - > 0.1 - for: 5m - labels: - severity: warning - annotations: - summary: "Workflow {{ $labels.workflow }} failure rate is {{ $value | humanizePercentage }}" -``` - -**Alert: AI connector latency exceeds 30 seconds:** - -```yaml - - alert: MantleSlowAIConnector - expr: | - histogram_quantile(0.95, rate(mantle_connector_duration_seconds_bucket{action="ai/completion"}[5m])) > 30 - for: 10m - labels: - severity: warning - annotations: - summary: "AI connector P95 latency is {{ $value }}s" -``` - -## Audit Trail - -Every state-changing operation in Mantle emits an audit event to the `audit_events` table in Postgres. The table is append-only -- events cannot be updated or deleted. This provides a tamper-resistant record of all activity. - -### What Gets Audited - -| Action | Trigger | Description | -|---|---|---| -| `workflow.applied` | `mantle apply` | A new workflow version was stored in the database. | -| `workflow.executed` | `mantle run`, cron, webhook, API | A workflow execution started. | -| `step.started` | Engine | A step began executing. | -| `step.completed` | Engine | A step finished successfully. | -| `step.failed` | Engine | A step failed (error is recorded). | -| `step.skipped` | Engine | A step was skipped because its `if` condition evaluated to `false`. | -| `execution.cancelled` | `mantle cancel`, API | An execution was cancelled. | - -### Audit Event Structure - -Each audit event contains: - -| Field | Description | -|---|---| -| `id` | Unique event identifier. | -| `timestamp` | When the event occurred (UTC). | -| `actor` | Who performed the action: `cli`, `engine`, `scheduler`, or a user ID. | -| `action` | The action type (e.g., `workflow.applied`). | -| `resource_type` | The type of resource affected (e.g., `workflow_definition`, `workflow_execution`, `step_execution`). | -| `resource_id` | The identifier of the affected resource. | -| `before_state` | Optional JSON: the state of the resource before the change. | -| `after_state` | Optional JSON: the state of the resource after the change. | -| `metadata` | Optional key-value pairs with additional context. | - -### Querying Audit Events - -Use the `mantle audit` command to query audit events from the command line: - -```bash -# All recent events (default: last 50) -mantle audit - -# Filter by action type -mantle audit --action workflow.applied - -# Filter by actor -mantle audit --actor cli - -# Filter by resource -mantle audit --resource workflow_definition/hello-world - -# Filter by time range -mantle audit --since 24h - -# Combine filters -mantle audit --action step.failed --since 7d --limit 100 -``` - -Output format: - -``` -2026-03-18T14:30:00Z cli workflow.applied workflow_definition/hello-world -2026-03-18T14:30:01Z engine workflow.executed workflow_execution/a1b2c3d4 -2026-03-18T14:30:01Z engine step.started step_execution/fetch -2026-03-18T14:30:02Z engine step.completed step_execution/fetch -``` - -### Querying Audit Events via SQL - -For advanced queries, you can query the `audit_events` table directly: - -```sql --- Find all failed steps in the last 24 hours -SELECT timestamp, actor, action, resource_type, resource_id -FROM audit_events -WHERE action = 'step.failed' - AND timestamp >= NOW() - INTERVAL '24 hours' -ORDER BY timestamp DESC; - --- Count events by action type -SELECT action, COUNT(*) as count -FROM audit_events -WHERE timestamp >= NOW() - INTERVAL '7 days' -GROUP BY action -ORDER BY count DESC; -``` - -### Audit Retention - -Mantle does not automatically delete old audit events. In a long-running production deployment, you may want to set up a retention policy. Example using a Postgres cron job (via `pg_cron`): - -```sql --- Delete audit events older than 90 days -DELETE FROM audit_events WHERE timestamp < NOW() - INTERVAL '90 days'; -``` - -Run this as a scheduled task outside of Mantle. - -## Structured JSON Logging - -In server mode, Mantle emits structured JSON logs to stdout via Go's `slog` package. Each log line is a self-contained JSON object. - -### Log Format - -```json -{"time":"2026-03-18T14:30:00.000Z","level":"INFO","msg":"server listening","address":":8080"} -{"time":"2026-03-18T14:30:01.000Z","level":"INFO","msg":"cron scheduler started"} -{"time":"2026-03-18T14:30:02.000Z","level":"INFO","msg":"workflow execution started","workflow":"hello-world","execution_id":"abc123"} -{"time":"2026-03-18T14:30:03.000Z","level":"ERROR","msg":"step failed","workflow":"hello-world","step":"fetch","error":"connection refused"} -``` - -### Log Levels - -| Level | Description | -|---|---| -| `debug` | Detailed diagnostic information. Includes connector request/response details and CEL expression evaluation. | -| `info` | Normal operational events. Server startup, workflow executions, cron triggers. | -| `warn` | Abnormal situations that do not prevent operation. Shutdown timeouts, deprecated features. | -| `error` | Errors that caused a step or operation to fail. | - -Configure the level with: - -```bash -# CLI flag -mantle serve --log-level debug - -# Environment variable -export MANTLE_LOG_LEVEL=warn - -# Config file -# mantle.yaml -log: - level: info -``` - -### Integration with Log Aggregation - -The JSON log format integrates directly with standard log aggregation systems. - -**Datadog:** - -Datadog's agent automatically parses JSON logs from stdout. If running Mantle in a container, configure the Datadog agent to collect container logs: - -```yaml -# datadog.yaml (agent config) -logs: - - type: docker - service: mantle - source: go -``` - -**ELK Stack (Elasticsearch, Logstash, Kibana):** - -Configure Filebeat to read Mantle's stdout (typically via container logs or a file redirect): - -```yaml -# filebeat.yml -filebeat.inputs: - - type: container - paths: - - /var/log/containers/mantle-*.log - json.keys_under_root: true - json.add_error_key: true -``` - -**Grafana Loki:** - -Use Promtail to ship logs from Mantle containers to Loki: - -```yaml -# promtail.yaml -scrape_configs: - - job_name: mantle - static_configs: - - targets: [localhost] - labels: - job: mantle - __path__: /var/log/mantle/*.log - pipeline_stages: - - json: - expressions: - level: level - msg: msg - - labels: - level: -``` - -**CloudWatch Logs:** - -When running on ECS or EKS, stdout logs are automatically captured by the `awslogs` log driver. No additional configuration is needed. - -### Correlating Logs with Metrics and Audit Events - -All three observability signals share common identifiers: - -| Identifier | In Logs | In Metrics | In Audit Events | -|---|---|---|---| -| Workflow name | `workflow` field | `workflow` label | `resource_id` (for workflow resources) | -| Execution ID | `execution_id` field | -- | `resource_id` (for execution resources) | -| Step name | `step` field | `step` label | `resource_id` (for step resources) | - -To trace a specific execution: - -1. Find the execution in `mantle logs ` or `GET /api/v1/executions/{id}` -2. Search structured logs for `execution_id` to see detailed step-level events -3. Query audit events with `mantle audit --resource workflow_execution/` for the audit trail -4. Check Prometheus metrics for timing data with the `workflow` label - -## REST API for Execution Queries - -In addition to the CLI and Prometheus metrics, you can query execution data through the REST API. This is useful for building custom dashboards or integrating with external monitoring systems. - -```bash -# List recent executions -curl -s "http://localhost:8080/api/v1/executions?limit=10" | jq . - -# Get details for a specific execution -curl -s "http://localhost:8080/api/v1/executions/abc123-def456" | jq . - -# Filter by workflow and status -curl -s "http://localhost:8080/api/v1/executions?workflow=daily-report&status=failed&since=24h" | jq . -``` - -See the [CLI Reference](cli-reference.md#rest-api) for the full API specification. - -## Further Reading - -- [CLI Reference](cli-reference.md#mantle-audit) -- `mantle audit` command and flags -- [CLI Reference](cli-reference.md#rest-api) -- REST API endpoint documentation -- [Server Guide](server-guide.md) -- running `mantle serve` with health probes and metrics -- [Concepts](concepts.md#observability) -- architectural overview of the observability stack -- [Configuration](configuration.md) -- log level and server address configuration diff --git a/docs/plugins-guide.md b/docs/plugins-guide.md deleted file mode 100644 index a04443e..0000000 --- a/docs/plugins-guide.md +++ /dev/null @@ -1,329 +0,0 @@ -# Plugins Guide - -Plugins extend Mantle with third-party connector actions. A plugin is an executable binary that communicates with the engine over a JSON stdin/stdout protocol. Plugins run as subprocesses -- they cannot access the engine's memory, database, or internal state directly. - -This guide covers what plugins are, how to write one, and how to install and manage them. - -## When to Use a Plugin - -Use a plugin when you need a connector action that Mantle does not provide out of the box. The built-in connectors cover HTTP, AI/LLM, Slack, Postgres, Email, and S3. For anything else -- a proprietary API, a custom data transformation, a niche SaaS integration -- write a plugin. - -Plugins are the right choice when: - -- You need to call an API that requires custom authentication or request formatting -- You want to reuse a connector across multiple Mantle installations -- You need to keep proprietary integration logic separate from the open-source core - -## How Plugins Work - -When a workflow step references a plugin action, the engine: - -1. Looks up the plugin binary in the `.mantle/plugins/` directory -2. Spawns it as a subprocess -3. Writes a JSON request to the plugin's stdin -4. Reads the JSON response from the plugin's stdout -5. Terminates the subprocess - -Each step execution spawns a fresh process. There is no persistent connection or shared state between invocations. - -``` -Engine Plugin Process - | | - |-- spawn subprocess ------------>| - |-- write JSON to stdin --------->| - | |-- parse input - | |-- execute action - | |-- write JSON to stdout - |<-- read JSON from stdout -------| - |-- subprocess exits ------------>| -``` - -The plugin has a 60-second timeout by default. If it does not produce output within that window, the engine kills the process and fails the step. - -## The JSON Protocol - -### Input (stdin) - -The engine writes a single JSON object to the plugin's stdin: - -```json -{ - "action": "my-plugin/fetch-data", - "params": { - "url": "https://api.example.com/data", - "limit": 100 - }, - "credential": { - "api_key": "sk-abc123" - } -} -``` - -| Field | Type | Description | -|---|---|---| -| `action` | string | The full action name from the workflow step (e.g., `my-plugin/fetch-data`). | -| `params` | object | The `params` map from the workflow step, with CEL expressions already evaluated. | -| `credential` | object | Decrypted credential fields, if the step has a `credential` reference. Empty object if no credential. | - -### Output (stdout) - -The plugin writes a single JSON object to stdout: - -```json -{ - "result": "success", - "items": [ - {"id": 1, "name": "Item A"}, - {"id": 2, "name": "Item B"} - ] -} -``` - -The output object becomes the step's `output` in subsequent CEL expressions. For example, `steps['my-step'].output.items[0].name` evaluates to `"Item A"`. - -### Errors - -If the plugin encounters an error, it should write a message to stderr and exit with a non-zero exit code. The engine captures stderr and reports it as the step error: - -``` -step.failed: plugin "my-plugin" failed: API returned 403 Forbidden -``` - -Do not write error details to stdout -- the engine only parses stdout as the output object. - -## Writing a Plugin - -A plugin can be written in any language. The only requirements are: - -1. It is an executable binary (or a script with a shebang line) -2. It reads JSON from stdin -3. It writes JSON to stdout -4. It exits with code 0 on success or non-zero on failure - -### Example: Python Plugin - -This minimal plugin fetches data from a custom API: - -```python -#!/usr/bin/env python3 -"""my-api-connector — a Mantle plugin for the Example API.""" - -import json -import sys -import urllib.request - -def main(): - # Read input from stdin. - raw = sys.stdin.read() - request = json.loads(raw) - - action = request["action"] - params = request["params"] - credential = request.get("credential", {}) - - if action != "example-api/fetch": - print(f"Unknown action: {action}", file=sys.stderr) - sys.exit(1) - - # Build the API request. - url = params["url"] - api_key = credential.get("api_key", "") - - req = urllib.request.Request(url) - req.add_header("Authorization", f"Bearer {api_key}") - - try: - with urllib.request.urlopen(req, timeout=30) as resp: - body = json.loads(resp.read()) - except Exception as e: - print(f"Request failed: {e}", file=sys.stderr) - sys.exit(1) - - # Write output to stdout. - output = { - "status": resp.status, - "data": body, - } - json.dump(output, sys.stdout) - -if __name__ == "__main__": - main() -``` - -Make it executable: - -```bash -chmod +x my-api-connector -``` - -### Example: Go Plugin - -```go -package main - -import ( - "encoding/json" - "fmt" - "os" -) - -type Input struct { - Action string `json:"action"` - Params map[string]any `json:"params"` - Credential map[string]string `json:"credential"` -} - -func main() { - var input Input - if err := json.NewDecoder(os.Stdin).Decode(&input); err != nil { - fmt.Fprintf(os.Stderr, "failed to parse input: %s\n", err) - os.Exit(1) - } - - // Implement your connector logic here. - output := map[string]any{ - "result": "ok", - "action": input.Action, - } - - if err := json.NewEncoder(os.Stdout).Encode(output); err != nil { - fmt.Fprintf(os.Stderr, "failed to write output: %s\n", err) - os.Exit(1) - } -} -``` - -Build and install: - -```bash -go build -o my-go-connector . -mantle plugins install ./my-go-connector -``` - -### Example: Shell Script Plugin - -For quick prototyping, a shell script works: - -```bash -#!/bin/bash -# Read the full JSON input. -INPUT=$(cat) - -# Extract fields with jq. -ACTION=$(echo "$INPUT" | jq -r '.action') -URL=$(echo "$INPUT" | jq -r '.params.url') - -# Do the work. -RESPONSE=$(curl -sf "$URL") -if [ $? -ne 0 ]; then - echo "HTTP request failed" >&2 - exit 1 -fi - -# Write JSON output. -echo "{\"body\": $RESPONSE}" -``` - -## Installing and Managing Plugins - -### Install - -Copy a plugin binary into the plugin directory: - -```bash -mantle plugins install ./path/to/my-plugin -``` - -This copies the file to `.mantle/plugins/my-plugin`. The plugin name is derived from the filename. - -### List - -See all installed plugins: - -```bash -mantle plugins list -``` - -### Remove - -Remove a plugin by name: - -```bash -mantle plugins remove my-plugin -``` - -This deletes the binary from the plugin directory. - -### Plugin Directory - -Plugins are stored in `.mantle/plugins/` relative to the current working directory. The directory is created automatically when you install the first plugin. - -``` -.mantle/ - plugins/ - my-api-connector - my-go-connector - data-transformer -``` - -## Using a Plugin in a Workflow - -Reference the plugin action in a step's `action` field. The action name is `/`: - -```yaml -name: custom-integration -steps: - - name: fetch-external-data - action: example-api/fetch - credential: my-api-key - timeout: "30s" - params: - url: "https://api.example.com/data" - limit: 100 - - - name: process-data - action: ai/completion - credential: my-openai - params: - model: gpt-4o - prompt: "Summarize: {{ steps['fetch-external-data'].output.data }}" -``` - -Plugins work with all standard step features: `if` conditions, `retry`, `timeout`, and `credential` resolution. - -## The Protobuf Specification - -The formal plugin contract is defined in `proto/connector.proto`. While the current implementation uses JSON stdin/stdout, the protobuf definition serves as the specification for a future gRPC-based protocol. - -The service defines three RPCs: - -```protobuf -service Connector { - rpc Execute(ExecuteRequest) returns (ExecuteResponse); - rpc Validate(ValidateRequest) returns (ValidateResponse); - rpc Describe(DescribeRequest) returns (DescribeResponse); -} -``` - -- **Execute** -- runs the action with parameters and credentials. This is the only RPC that the JSON protocol currently implements. -- **Validate** -- checks whether parameters are valid without executing. Planned for a future version. -- **Describe** -- returns metadata about the plugin's supported actions. Planned for a future version. - -## Best Practices - -- **Keep plugins stateless.** Each invocation is a fresh process. Do not rely on files, environment variables, or other side effects from previous runs. -- **Validate input early.** Check for required params and credential fields before doing any work. Exit with a clear error message on stderr. -- **Set timeouts on external calls.** The engine applies a 60-second timeout to the subprocess, but your plugin should set its own timeouts on network requests to fail gracefully. -- **Test with stdin/stdout directly.** You can test a plugin without Mantle by piping JSON: - -```bash -echo '{"action":"my-plugin/fetch","params":{"url":"https://example.com"},"credential":{}}' | ./my-plugin -``` - -- **Keep output small.** The entire stdout output is stored as the step's output in the database checkpoint. Avoid returning megabytes of data -- filter and summarize in the plugin. - -## Further Reading - -- [CLI Reference](cli-reference.md#mantle-plugins) -- `mantle plugins` command documentation -- [Workflow Reference](workflow-reference.md#connectors) -- connector actions and the `action` field -- [Concepts](concepts.md#plugin-system) -- architectural overview of the plugin system diff --git a/docs/secrets-guide.md b/docs/secrets-guide.md deleted file mode 100644 index 5c33004..0000000 --- a/docs/secrets-guide.md +++ /dev/null @@ -1,411 +0,0 @@ -# Secrets Management Guide - -Mantle includes a built-in secrets system for storing API keys, tokens, and other credentials. Credentials are encrypted at rest and resolved at connector invocation time -- they are never exposed in CEL expressions, workflow logs, or step outputs. - -This guide covers setting up encryption, creating and managing credentials, and using them in workflows. - -## Why Secrets Exist - -Hardcoding API keys in workflow YAML files is a security risk. Anyone with access to the file sees the raw key. Mantle solves this with typed credentials that are: - -- **Encrypted at rest** using AES-256-GCM -- **Resolved at runtime** inside the connector, not in the expression engine -- **Never logged** -- credential values do not appear in execution logs or step outputs - -You reference credentials by name in your workflow YAML. The engine resolves the name to decrypted field values at the moment a connector needs them. - -## Setting Up the Encryption Key - -The secrets system requires a 32-byte hex-encoded encryption key. You configure it in one of two ways: - -**Environment variable (recommended for production):** - -```bash -export MANTLE_ENCRYPTION_KEY="your-64-char-hex-string-here" -``` - -**Config file:** - -```yaml -# mantle.yaml -encryption: - key: "your-64-char-hex-string-here" -``` - -The encryption key is required for all `mantle secrets` commands. Other Mantle commands (like `validate`, `apply`, and `run`) work without it -- you only need the key when creating, listing, deleting, or rotating credentials, or when running workflows that reference credentials. - -### Generating a Key - -Generate a cryptographically secure 32-byte key with `openssl`: - -```bash -openssl rand -hex 32 -``` - -This outputs a 64-character hex string suitable for `MANTLE_ENCRYPTION_KEY`. - -Alternatively, the `mantle secrets rotate-key` command auto-generates a new key if you do not provide one (see [Key Rotation](#key-rotation) below). - -## Creating Credentials - -Use `mantle secrets create` to store a new credential: - -```bash -mantle secrets create --name my-openai --type openai --field api_key=sk-proj-abc123 -``` - -The `--field` flag is repeatable. Pass one `--field key=value` per credential field. - -**Example with multiple fields:** - -```bash -mantle secrets create --name my-openai --type openai \ - --field api_key=sk-proj-abc123 \ - --field org_id=org-xyz789 -``` - -**Example with basic auth:** - -```bash -mantle secrets create --name my-api --type basic \ - --field username=admin \ - --field password=s3cret -``` - -On success, you see: - -``` -Created credential "my-openai" (type: openai) -``` - -## Credential Types - -Each credential type has a defined set of fields. Mantle validates that all required fields are present when you create a credential. - -| Type | Required Fields | Optional Fields | Use Case | -|---|---|---|---| -| `generic` | `key` | -- | General-purpose API key or secret value | -| `bearer` | `token` | -- | Bearer token authentication | -| `openai` | `api_key` | `org_id` | OpenAI API access | -| `basic` | `username`, `password` | -- | HTTP Basic authentication | - -If you pass a field name that does not belong to the credential type, it is silently ignored. If you omit a required field, you get an error: - -``` -Error: field "api_key" is required for credential type "openai" -``` - -## Listing Credentials - -List all stored credentials with `mantle secrets list`: - -```bash -$ mantle secrets list -NAME TYPE CREATED -my-openai openai 2026-03-18 14:30:00 -my-api basic 2026-03-18 14:35:00 -``` - -This command shows name, type, and creation date. It never displays decrypted values. - -If no credentials exist: - -``` -(no credentials) -``` - -## Deleting Credentials - -Delete a credential by name: - -```bash -$ mantle secrets delete --name my-openai -Deleted credential "my-openai" -``` - -Deleting a credential that is referenced by a workflow does not affect stored workflow definitions. The workflow fails at execution time if the credential cannot be resolved. - -## Using Credentials in Workflows - -Reference a stored credential by adding the `credential` field to a step: - -```yaml -steps: - - name: call-openai - action: ai/completion - credential: my-openai - params: - model: gpt-4o - prompt: "Summarize this text: {{ steps.fetch-data.output.body }}" -``` - -When the engine reaches this step, it resolves `my-openai` to its decrypted field values and passes them to the connector. For the `openai` credential type, the AI connector uses the `api_key` field for the `Authorization` header and the `org_id` field (if present) for the `OpenAI-Organization` header. - -The `credential` field works on any step, not just AI steps. For example, you can use a `bearer` credential with an HTTP request: - -```yaml -steps: - - name: fetch-data - action: http/request - credential: my-api-token - params: - method: GET - url: https://api.example.com/data -``` - -### Credential Resolution Order - -When a step references a credential, the engine resolves it in this order: - -1. **Postgres credentials table** -- looks up the credential by name and decrypts it -2. **Cloud secret backends** -- tries each configured backend in registration order (AWS, GCP, Azure) -3. **Environment variable fallback** -- checks for `MANTLE_SECRET_` - -The first source to return a value wins. If the credential is not found in any source, the command fails with an error listing all sources that were checked. - -The environment variable name is derived from the credential name by converting to uppercase and replacing hyphens with underscores. For example: - -| Credential Name | Env Var Fallback | -|---|---| -| `my-openai` | `MANTLE_SECRET_MY_OPENAI` | -| `my_api_key` | `MANTLE_SECRET_MY_API_KEY` | -| `prod-token` | `MANTLE_SECRET_PROD_TOKEN` | - -The env var fallback returns the value as a single `key` field, equivalent to a `generic` credential. This is useful for local development or CI where you do not want to set up the full encryption pipeline. - -## Cloud Secret Backends - -Mantle can resolve credentials from external cloud secret stores in addition to the local Postgres table. This lets you centralize secret management across your infrastructure. - -All cloud backends share the same behavior: if the secret value in the cloud store is valid JSON that decodes to a flat `map[string]string`, those key-value pairs are returned directly as credential fields. If the value is an opaque string, it is returned as `{"key": ""}`, equivalent to a `generic` credential. - -### AWS Secrets Manager - -Mantle resolves secrets from AWS Secrets Manager using the standard AWS SDK credential chain (environment variables, shared config, EC2/ECS instance profile). - -**IAM permissions required:** - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": "secretsmanager:GetSecretValue", - "Resource": "arn:aws:secretsmanager:*:*:secret:*" - } - ] -} -``` - -Scope the `Resource` field to limit access to specific secrets. - -**Secret naming:** The credential name in your workflow maps 1:1 to the secret name in AWS Secrets Manager. If your workflow step references `credential: my-openai`, Mantle calls `GetSecretValue` with `SecretId: "my-openai"`. - -**Storing structured credentials in AWS:** - -Store the secret as a JSON object to provide multiple fields: - -```bash -aws secretsmanager create-secret \ - --name my-openai \ - --secret-string '{"api_key":"sk-proj-abc123","org_id":"org-xyz789"}' -``` - -The connector receives the parsed fields `api_key` and `org_id`, the same as a Postgres-stored `openai` credential. - -**Storing simple values:** - -```bash -aws secretsmanager create-secret \ - --name my-api-key \ - --secret-string "sk-proj-abc123" -``` - -This is returned as `{"key": "sk-proj-abc123"}`. - -**Setup:** Configure the AWS region through the standard `AWS_REGION` or `AWS_DEFAULT_REGION` environment variables. The backend uses `config.LoadDefaultConfig`, so all standard AWS credential sources are supported. - -### GCP Secret Manager - -Mantle resolves secrets from GCP Secret Manager using Application Default Credentials (ADC). - -**IAM permissions required:** - -The service account or user needs the `secretmanager.versions.access` permission on the target secrets. The simplest way is to grant the `roles/secretmanager.secretAccessor` role: - -```bash -gcloud secrets add-iam-policy-binding my-openai \ - --member="serviceAccount:mantle@my-project.iam.gserviceaccount.com" \ - --role="roles/secretmanager.secretAccessor" -``` - -**Secret naming:** The credential name maps to the secret name in GCP. Mantle constructs the resource path as `projects/{project_id}/secrets/{name}/versions/latest`, always fetching the latest version. - -**GCP project ID:** The backend requires a GCP project ID. Configure this through the `MANTLE_GCP_PROJECT` environment variable or the ADC project setting. - -**Storing credentials in GCP:** - -```bash -echo -n '{"api_key":"sk-proj-abc123","org_id":"org-xyz789"}' | \ - gcloud secrets create my-openai --data-file=- -``` - -**Authentication setup:** Configure ADC through one of these methods: - -- `gcloud auth application-default login` (local development) -- Workload Identity on GKE -- Service account key file (set `GOOGLE_APPLICATION_CREDENTIALS`) - -### Azure Key Vault - -Mantle resolves secrets from Azure Key Vault using the DefaultAzureCredential chain (environment variables, managed identity, Azure CLI, etc.). - -**Required permissions:** The identity needs the `Get` secret permission on the Key Vault. Assign it through Azure RBAC or Key Vault access policies: - -```bash -az keyvault set-policy --name my-vault \ - --object-id \ - --secret-permissions get -``` - -**Vault URL:** Configure the Key Vault URL through the `MANTLE_AZURE_VAULT_URL` environment variable (e.g., `https://my-vault.vault.azure.net/`). - -**Secret naming:** The credential name maps 1:1 to the secret name in Key Vault. Mantle always fetches the latest version. - -**Storing credentials in Azure:** - -```bash -az keyvault secret set --vault-name my-vault \ - --name my-openai \ - --value '{"api_key":"sk-proj-abc123","org_id":"org-xyz789"}' -``` - -**Authentication setup:** Configure DefaultAzureCredential through one of these methods: - -- `az login` (local development) -- Managed Identity on Azure VMs, App Service, or AKS -- Environment variables: `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_SECRET` - -### Resolution Order with Cloud Backends - -When cloud backends are configured, the full resolution order is: - -``` -1. Postgres store (if encryption key is configured) -2. AWS Secrets Manager (if AWS credentials are available) -3. GCP Secret Manager (if GCP project is configured) -4. Azure Key Vault (if vault URL is configured) -5. Environment variable fallback (MANTLE_SECRET_) -``` - -Cloud backends are tried in the order they are registered. The engine falls through to the next source on any error (not found, permission denied, network timeout). This means a credential stored in both Postgres and AWS resolves from Postgres because it is checked first. - -**When to use cloud backends vs. Postgres:** - -| Scenario | Recommendation | -|---|---| -| Single-instance deployment | Postgres store is simplest | -| Secrets shared across multiple services | Cloud backend (centralized management) | -| Compliance requirement for a specific vault | Cloud backend | -| Local development | Environment variable fallback | - -## Key Rotation - -If your encryption key is compromised or your security policy requires periodic rotation, use `mantle secrets rotate-key` to re-encrypt all credentials with a new key. - -### How Rotation Works - -The `mantle secrets rotate-key` command decrypts every credential in the `credentials` table using the current (old) key, then re-encrypts each one with the new key. The entire operation runs inside a single Postgres transaction -- either all credentials are re-encrypted or none are. If any credential fails to decrypt (e.g., corrupted data), the transaction is rolled back and no changes are written. - -### When to Rotate - -Rotate your encryption key when: - -- **Key compromise** -- the current key has been exposed in logs, version control, or an unauthorized system -- **Personnel change** -- an employee or contractor with access to the key has left the organization -- **Policy compliance** -- your security policy requires periodic key rotation (e.g., every 90 days) -- **Environment migration** -- you are moving from a development key to a production key - -### Step-by-Step Procedure - -**1. Generate a new 32-byte encryption key:** - -```bash -openssl rand -hex 32 -``` - -Save the output -- this is your new 64-character hex key. - -**2. Run the rotation command:** - -You can provide the old key explicitly with `--old-key`, which is useful when the new key is already set in the environment: - -```bash -MANTLE_ENCRYPTION_KEY= mantle secrets rotate-key --old-key -``` - -Or let Mantle use the current `MANTLE_ENCRYPTION_KEY` as the old key and provide the new key with `--new-key`: - -```bash -mantle secrets rotate-key --new-key -``` - -Or omit `--new-key` entirely to have Mantle auto-generate a new key: - -```bash -$ mantle secrets rotate-key -Re-encrypted 3 credential(s). -New key: a1b2c3d4e5f6... -Update MANTLE_ENCRYPTION_KEY to the new key and restart. -``` - -**3. Update your configuration with the new key:** - -Replace `MANTLE_ENCRYPTION_KEY` in your environment, secret store, or `mantle.yaml` with the new key. If running in Kubernetes, update the Helm values or Kubernetes secret: - -```bash -kubectl create secret generic mantle-encryption \ - --from-literal=key= \ - --dry-run=client -o yaml | kubectl apply -f - -``` - -Then restart the Mantle process so it picks up the new key. - -**4. Verify the rotation:** - -```bash -mantle secrets list -``` - -This should display all credentials with their names and types. If the key is wrong, the command fails with a decryption error. - -### Auto-Generate vs. Explicit Key - -| Method | Command | When to Use | -|---|---|---| -| Auto-generate | `mantle secrets rotate-key` | Quick rotation; Mantle prints the new key | -| Explicit new key | `mantle secrets rotate-key --new-key ` | You pre-generated the key or it comes from a vault | -| Explicit old key | `MANTLE_ENCRYPTION_KEY= mantle secrets rotate-key --old-key ` | Environment already has the new key set | - -### Important Notes - -- The rotation is **transactional** -- all credentials are re-encrypted atomically. If the process is interrupted, no partial state is left behind. -- After rotation, the old key is no longer valid. Any Mantle instance still configured with the old key will fail to decrypt credentials. -- Rotation does not affect credentials stored in cloud secret backends (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault). Only Postgres-stored credentials are re-encrypted. -- There is no downtime required. Run the rotation, update the key in your configuration, and restart the server. - -## Security Model - -- **Encryption algorithm**: AES-256-GCM (authenticated encryption with associated data) -- **Key storage**: The encryption key is not stored in the database. You manage it through environment variables or the config file. -- **At rest**: All credential field values are encrypted before being written to Postgres. The credential name and type are stored in plaintext for listing and lookup. -- **In expressions**: Credentials are never available in CEL expressions. You cannot reference `steps.my-step.credential` or access decrypted values through template interpolation. The credential is resolved inside the connector implementation. -- **In logs**: Credential values never appear in execution logs, step outputs, or error messages. - -## Further Reading - -- [CLI Reference](cli-reference.md) -- full flag documentation for all secrets commands -- [Workflow Reference](workflow-reference.md) -- the `credential` field on steps and all connectors -- [Configuration](configuration.md) -- setting `encryption.key`, `MANTLE_ENCRYPTION_KEY`, and cloud backend env vars -- [Concepts](concepts.md) -- how credential resolution fits into the engine architecture diff --git a/docs/server-guide.md b/docs/server-guide.md deleted file mode 100644 index ae69b15..0000000 --- a/docs/server-guide.md +++ /dev/null @@ -1,514 +0,0 @@ -# Server Guide - -This guide covers running Mantle as a persistent server with automatic triggers, the REST API, and production deployment. For the YAML trigger syntax, see the [Workflow Reference](workflow-reference.md#triggers). For the `mantle serve` command flags, see the [CLI Reference](cli-reference.md#mantle-serve). - -## Starting the Server - -Run `mantle serve` to start Mantle as a long-running process: - -```bash -mantle serve -``` - -The server: - -1. Runs all pending database migrations -2. Starts the HTTP API, webhook listener, and health endpoints on the configured address (default `:8080`) -3. Starts the cron scheduler, polling every 30 seconds for due triggers - -The server keeps running until you stop it with `SIGTERM` or `SIGINT` (Ctrl+C). - -### Custom Listen Address - -Use the `--api-address` flag or `MANTLE_API_ADDRESS` environment variable to change the listen address: - -```bash -mantle serve --api-address :9090 -``` - -See [Configuration](configuration.md) for all configuration options. - -## Setting Up Cron Triggers - -A cron trigger executes a workflow on a recurring schedule. Add a `triggers` section to your workflow YAML: - -```yaml -name: daily-report -description: Generate and email a daily summary report - -triggers: - - type: cron - schedule: "0 9 * * 1-5" - -steps: - - name: fetch-metrics - action: http/request - params: - method: GET - url: https://api.internal.com/metrics/daily - - - name: summarize - action: ai/completion - credential: my-openai - params: - model: gpt-4o - prompt: "Summarize these metrics into 5 bullet points: {{ steps.fetch-metrics.output.body }}" -``` - -Apply the workflow to register the trigger: - -```bash -mantle apply daily-report.yaml -# Applied daily-report version 1 -``` - -The cron scheduler picks up the trigger the next time it polls (within 30 seconds). The workflow runs every weekday at 9 AM. - -### Cron Expression Syntax - -The `schedule` field uses standard 5-field cron syntax: - -``` -┌───────────── minute (0-59) -│ ┌───────────── hour (0-23) -│ │ ┌───────────── day of month (1-31) -│ │ │ ┌───────────── month (1-12) -│ │ │ │ ┌───────────── day of week (0-6, Sunday=0) -│ │ │ │ │ -* * * * * -``` - -Common patterns: - -| Schedule | Expression | -|---|---| -| Every minute | `* * * * *` | -| Every 5 minutes | `*/5 * * * *` | -| Every hour on the hour | `0 * * * *` | -| Daily at midnight | `0 0 * * *` | -| Weekdays at 9 AM | `0 9 * * 1-5` | -| First of every month at noon | `0 12 1 * *` | -| Every 15 minutes during business hours | `*/15 9-17 * * 1-5` | - -### Updating a Cron Schedule - -Edit the `schedule` field in your YAML and re-apply: - -```bash -# Change from every 5 minutes to every 15 minutes -mantle apply daily-report.yaml -# Applied daily-report version 2 -``` - -The scheduler picks up the updated schedule on the next poll cycle. - -### Removing a Cron Trigger - -Delete the `triggers` section from the YAML (or remove the specific trigger entry) and re-apply: - -```bash -mantle apply daily-report.yaml -# Applied daily-report version 3 -``` - -The trigger is deregistered. The workflow is still available for manual execution with `mantle run` or the REST API. - -## Setting Up Webhook Triggers - -A webhook trigger executes a workflow when an HTTP POST request arrives at a configured path. The request body is available as `trigger.payload` in CEL expressions. - -```yaml -name: deploy-notifier -description: Post a Slack notification when a deploy completes - -triggers: - - type: webhook - path: "/hooks/deploy-notifier" - -steps: - - name: notify-slack - action: http/request - params: - method: POST - url: https://hooks.slack.com/services/T00/B00/xxx - body: - text: "Deployed {{ trigger.payload.repo }}@{{ trigger.payload.sha }} to {{ trigger.payload.environment }}" -``` - -Apply the workflow: - -```bash -mantle apply deploy-notifier.yaml -# Applied deploy-notifier version 1 -``` - -Trigger it from your CI pipeline or any HTTP client: - -```bash -curl -X POST http://localhost:8080/hooks/deploy-notifier \ - -H "Content-Type: application/json" \ - -d '{ - "repo": "my-app", - "sha": "abc1234", - "environment": "production" - }' -``` - -The server starts a new execution and the full JSON body is available as `trigger.payload`. - -### Accessing Webhook Payload Data - -The `trigger.payload` variable contains the parsed JSON body. Access nested fields with dot notation in template strings or bracket notation in `if` expressions: - -```yaml -# In template strings (params): -url: "{{ trigger.payload.callback_url }}" -prompt: "Analyze this event: {{ trigger.payload }}" - -# In if expressions: -if: "trigger.payload.action == 'opened'" -``` - -## Calling the REST API - -The server exposes a REST API for triggering and managing executions programmatically. - -### Trigger a Workflow - -``` -POST /api/v1/run/{workflow} -``` - -Starts a new execution of the named workflow using the latest applied version. - -**Request:** - -```bash -curl -s -X POST http://localhost:8080/api/v1/run/fetch-and-summarize | jq . -``` - -**Response:** - -```json -{ - "execution_id": "abc123-def456", - "workflow": "fetch-and-summarize", - "version": 2 -} -``` - -### Cancel an Execution - -``` -POST /api/v1/cancel/{execution} -``` - -Cancels a running or pending execution. Equivalent to `mantle cancel`. - -**Request:** - -```bash -curl -s -X POST http://localhost:8080/api/v1/cancel/abc123-def456 -``` - -### List Executions - -``` -GET /api/v1/executions -``` - -Returns a list of recent executions with optional filtering. Supports query parameters: `workflow`, `status`, `since`, `limit`. - -```bash -curl -s "http://localhost:8080/api/v1/executions?workflow=daily-report&status=completed&limit=10" | jq . -``` - -```json -{ - "executions": [ - { - "id": "abc123-def456", - "workflow": "daily-report", - "version": 2, - "status": "completed", - "started_at": "2026-03-18T09:00:00Z", - "completed_at": "2026-03-18T09:00:15Z" - } - ] -} -``` - -### Get Execution Details - -``` -GET /api/v1/executions/{id} -``` - -Returns a single execution with step-level detail. - -```bash -curl -s http://localhost:8080/api/v1/executions/abc123-def456 | jq . -``` - -```json -{ - "id": "abc123-def456", - "workflow": "daily-report", - "version": 2, - "status": "completed", - "started_at": "2026-03-18T09:00:00Z", - "completed_at": "2026-03-18T09:00:15Z", - "steps": [ - { - "name": "fetch-metrics", - "status": "completed", - "started_at": "2026-03-18T09:00:00Z", - "completed_at": "2026-03-18T09:00:05Z" - }, - { - "name": "summarize", - "status": "completed", - "started_at": "2026-03-18T09:00:05Z", - "completed_at": "2026-03-18T09:00:15Z" - } - ] -} -``` - -### Prometheus Metrics - -``` -GET /metrics -``` - -Returns Prometheus metrics in exposition format. Scrape this endpoint with Prometheus, Grafana Agent, or any compatible collector. See the [Observability Guide](observability-guide.md) for metric names and example PromQL queries. - -### Health Endpoints - -| Endpoint | Purpose | Returns 200 When | -|---|---|---| -| `GET /healthz` | Liveness probe | The process is running | -| `GET /readyz` | Readiness probe | The database connection is healthy and migrations are applied | - -```bash -curl http://localhost:8080/healthz -# 200 OK - -curl http://localhost:8080/readyz -# 200 OK (when database is connected) -``` - -## Graceful Shutdown - -When the server receives `SIGTERM` or `SIGINT`: - -1. The HTTP server stops accepting new connections -2. In-flight HTTP requests are allowed to complete -3. Running workflow executions finish their current step and checkpoint -4. The process exits with code 0 - -This makes Mantle safe to run behind Kubernetes, systemd, or any process manager that sends `SIGTERM` on shutdown. - -## Production Deployment - -### Using the Helm Chart - -The recommended way to run Mantle in production is with the included Helm chart. It configures health probes, resource limits, and environment variables: - -```bash -helm install mantle charts/mantle \ - --set database.url="postgres://mantle:secret@db.internal:5432/mantle?sslmode=require" \ - --set encryption.key="your-hex-key" -``` - -The Helm chart configures: - -- Liveness probe pointing to `/healthz` -- Readiness probe pointing to `/readyz` -- `SIGTERM` as the termination signal (aligns with Mantle's graceful shutdown) - -### Health Probes - -Configure your load balancer or orchestrator to use the health endpoints: - -| Probe | Endpoint | Recommended Interval | -|---|---|---| -| Liveness | `GET /healthz` | 10s | -| Readiness | `GET /readyz` | 5s | - -The readiness probe returns a non-200 status when the database connection is lost, which causes the load balancer to stop routing traffic to the unhealthy instance. - -### Environment Variables - -In production, pass configuration through environment variables rather than config files: - -```bash -export MANTLE_DATABASE_URL="postgres://mantle:secret@db.internal:5432/mantle?sslmode=require" -export MANTLE_API_ADDRESS=":8080" -export MANTLE_ENCRYPTION_KEY="your-64-char-hex-key" -export MANTLE_LOG_LEVEL="warn" -mantle serve -``` - -See [Configuration](configuration.md) for the full list of environment variables. - -### Migrations - -The server runs migrations automatically on startup. You do not need a separate `mantle init` step in your deployment pipeline. This is safe to run with multiple replicas -- migrations use database-level locking to prevent conflicts. - -## Example: Workflow with Both Cron and Webhook Triggers - -This workflow monitors an API endpoint hourly and can also be triggered on demand by an external system: - -```yaml -name: api-health-check -description: Check API health and alert on failures - -triggers: - - type: cron - schedule: "0 * * * *" - - type: webhook - path: "/hooks/api-health-check" - -steps: - - name: check-api - action: http/request - timeout: 10s - retry: - max_attempts: 3 - backoff: exponential - params: - method: GET - url: https://api.example.com/health - - - name: alert-on-failure - action: http/request - if: "steps['check-api'].output.status_code != 200" - params: - method: POST - url: https://hooks.slack.com/services/T00/B00/xxx - body: - text: "API health check failed with status {{ steps.check-api.output.status_code }}" -``` - -Apply the workflow and start the server: - -```bash -mantle apply api-health-check.yaml -mantle serve -``` - -The workflow runs every hour automatically. You can also trigger it immediately: - -```bash -# Via the REST API -curl -X POST http://localhost:8080/api/v1/run/api-health-check - -# Via the webhook endpoint -curl -X POST http://localhost:8080/hooks/api-health-check -``` - -## Backup and Disaster Recovery - -Postgres is Mantle's single point of state. All workflow definitions, execution history, step checkpoints, encrypted credentials, and audit events live in the database. Recovery from any failure depends on having a good database backup and access to your encryption key. - -### What to Back Up - -| Asset | Location | Notes | -|---|---|---| -| Postgres database | Your database host | Contains all Mantle state: definitions, executions, credentials, audit events | -| Encryption key | `MANTLE_ENCRYPTION_KEY` env var or `mantle.yaml` | Required to decrypt credentials; store separately from database backups | -| `mantle.yaml` | Configuration file on disk | Can be reconstructed, but easier to back up | -| Workflow YAML files | Version control (Git) | Authoritative source; can be re-applied with `mantle apply` | - -**Critical:** Store the encryption key separately from database backups. If an attacker obtains both the database dump and the encryption key, they can decrypt all stored credentials. - -### Recommended Backup Approach - -**Managed Postgres (RDS, Cloud SQL, Azure Database):** - -- Enable automated daily snapshots with your cloud provider -- Enable WAL archiving (point-in-time recovery) for near-zero RPO -- Retain snapshots according to your compliance requirements - -**Self-hosted Postgres:** - -- Schedule `pg_dump` on a cron job (e.g., daily or hourly depending on RPO requirements): - -```bash -pg_dump -Fc -h localhost -U mantle mantle > /backups/mantle-$(date +%Y%m%d-%H%M%S).dump -``` - -- Configure WAL archiving for continuous point-in-time recovery -- Ship backups to off-site storage (S3, GCS, or another region) - -### Recovery Procedure - -1. **Restore Postgres from backup.** Use your cloud provider's restore flow or `pg_restore`: - -```bash -pg_restore -h localhost -U mantle -d mantle /backups/mantle-20260322-120000.dump -``` - -2. **Verify migration state.** Run `mantle init` to confirm all migrations are applied. If the backup is from an older version of Mantle, this applies any missing migrations: - -```bash -mantle init -``` - -3. **Verify credentials.** Confirm the encryption key matches the one used when the backup was taken: - -```bash -mantle secrets list -``` - -If this fails with a decryption error, the encryption key does not match the backup. - -4. **Resume the server:** - -```bash -mantle serve -``` - -5. **Re-apply workflow YAML files if needed.** Workflow definitions are stored in Postgres, so a database restore brings them back. However, if you need to apply changes that were made after the backup was taken, re-apply from version control: - -```bash -mantle apply workflows/*.yaml -``` - -### RPO and RTO Guidance - -| Backup Strategy | Recovery Point Objective (RPO) | Recovery Time Objective (RTO) | -|---|---|---| -| Daily `pg_dump` | Up to 24 hours of data loss | Minutes (restore + restart) | -| Hourly `pg_dump` | Up to 1 hour of data loss | Minutes | -| WAL archiving (PITR) | Near-zero (seconds of data loss) | Minutes to restore to a point in time | -| Managed snapshots + WAL | Near-zero | Depends on cloud provider restore time | - -For production deployments, WAL archiving combined with periodic base backups gives the best balance of RPO and operational simplicity. - -## Change Management - -All changes to Mantle's codebase and deployment follow a controlled process. - -1. **Pull request review.** All changes go through PR review on GitHub. At least one approval is required before merging to `main`. -2. **CI must pass.** Every PR runs the full CI pipeline: `go test`, `go vet`, `golangci-lint`, `govulncheck`, and `gosec`. PRs with failing checks are not merged. -3. **Production deployments.** Deploy via Helm with migration hooks. The Helm chart runs `mantle init` as a pre-install/pre-upgrade hook to apply database migrations before the new binary starts serving traffic. -4. **Rollback.** If a deployment introduces a problem, roll back with `helm rollback` and verify the migration state: - -```bash -helm rollback mantle -mantle migrate status -``` - -Mantle migrations are forward-only by design. Rolling back the binary to an older version is safe as long as the database schema is compatible. Check `mantle migrate status` to confirm. - -## Further Reading - -- [Workflow Reference](workflow-reference.md#triggers) -- trigger YAML syntax -- [CLI Reference](cli-reference.md#mantle-serve) -- `mantle serve` command flags -- [CLI Reference](cli-reference.md#rest-api) -- full REST API endpoint documentation -- [Configuration](configuration.md) -- all configuration options -- [Concepts](concepts.md#triggers-and-server-mode) -- architectural overview of triggers -- [Observability Guide](observability-guide.md) -- Prometheus metrics, audit trail, and structured logging -- [Secrets Guide](secrets-guide.md#key-rotation) -- encryption key rotation procedure diff --git a/docs/workflow-reference.md b/docs/workflow-reference.md deleted file mode 100644 index 829d756..0000000 --- a/docs/workflow-reference.md +++ /dev/null @@ -1,936 +0,0 @@ -# Workflow Reference - -This document describes every field in a Mantle workflow YAML file. For a hands-on introduction, start with the [Getting Started](getting-started.md) guide. - -## Complete Example - -```yaml -name: fetch-and-summarize -description: Fetch data from an API and summarize it with an LLM - -inputs: - url: - type: string - description: URL to fetch - max_retries: - type: number - description: Maximum number of retries for the HTTP request - -triggers: - - type: cron - schedule: "0 * * * *" - - type: webhook - path: "/hooks/fetch-and-summarize" - -steps: - - name: fetch-data - action: http/request - timeout: 30s - retry: - max_attempts: 3 - backoff: exponential - params: - method: GET - url: "{{ inputs.url }}" - - - name: summarize - action: ai/completion - timeout: 60s - params: - provider: openai - model: gpt-4o - prompt: "Summarize this data: {{ steps.fetch-data.output.body }}" - - - name: post-result - action: http/request - if: "steps.summarize.output.key_points.size() > 0" - params: - method: POST - url: https://hooks.example.com/results - body: - summary: "{{ steps.summarize.output.summary }}" -``` - -## Top-Level Fields - -| Field | Type | Required | Description | -|---|---|---|---| -| `name` | string | Yes | Unique identifier for the workflow. Must be kebab-case: lowercase letters, digits, and hyphens. Pattern: `^[a-z][a-z0-9-]*$`. | -| `description` | string | No | Human-readable description of what the workflow does. | -| `inputs` | map | No | Input parameters the workflow accepts at runtime. | -| `triggers` | list | No | Automatic triggers that start the workflow. See [Triggers](#triggers). | -| `steps` | list | Yes | Ordered list of steps to execute. At least one step is required. | - -### Name Rules - -The workflow name is the primary identifier used across `validate`, `apply`, `plan`, and `run`. It must: - -- Start with a lowercase letter -- Contain only lowercase letters (`a-z`), digits (`0-9`), and hyphens (`-`) -- Not start or end with a hyphen - -Valid examples: `fetch-data`, `my-workflow-v2`, `a1` - -Invalid examples: `Fetch-Data`, `fetch_data`, `-fetch`, `123abc` - -## Inputs - -Inputs define the parameters a workflow accepts when triggered. Each input is a key-value pair in the `inputs` map. - -```yaml -inputs: - url: - type: string - description: URL to fetch - verbose: - type: boolean - description: Enable verbose output - max_items: - type: number - description: Maximum number of items to process -``` - -### Input Fields - -| Field | Type | Required | Description | -|---|---|---|---| -| (key) | string | Yes | Input parameter name. Must be snake_case: lowercase letters, digits, and underscores. Pattern: `^[a-z][a-z0-9_]*$`. | -| `type` | string | No | Data type. One of: `string`, `number`, `boolean`. | -| `description` | string | No | Human-readable description. | - -### Input Name Rules - -Input names use snake_case (underscores), not kebab-case (hyphens). This is intentional -- input names appear in CEL expressions where hyphens would be interpreted as subtraction. - -Valid: `url`, `max_retries`, `api_key` - -Invalid: `URL`, `max-retries`, `apiKey`, `123abc` - -## Steps - -Steps are the building blocks of a workflow. Each step invokes a connector action and can optionally include conditional logic, retry policies, timeouts, and explicit dependencies. Steps without dependencies run concurrently; use `depends_on` to declare explicit ordering. See [Parallel Execution](#parallel-execution). - -```yaml -steps: - - name: fetch-data - action: http/request - timeout: 30s - retry: - max_attempts: 3 - backoff: exponential - if: "inputs.url != ''" - params: - method: GET - url: "{{ inputs.url }}" -``` - -### Step Fields - -| Field | Type | Required | Description | -|---|---|---|---| -| `name` | string | Yes | Unique name within the workflow. Must be kebab-case: `^[a-z][a-z0-9-]*$`. | -| `action` | string | Yes | Connector action to invoke, in `connector/action` format. | -| `params` | map | No | Parameters passed to the connector action. Structure depends on the action. | -| `if` | string | No | CEL expression. The step runs only if this evaluates to `true`. | -| `retry` | object | No | Retry policy for this step. See [Retry Policy](#retry-policy). | -| `timeout` | string | No | Maximum duration for the step. Uses Go duration format (e.g., `30s`, `5m`, `1h`). | -| `credential` | string | No | Name of a stored credential to inject into this step. See [Secrets Guide](secrets-guide.md). | -| `depends_on` | list of strings | No | Declares explicit dependencies on other steps for parallel execution. See [Parallel Execution](#parallel-execution). | - -### Step Name Rules - -Step names follow the same rules as the workflow name: kebab-case, starting with a lowercase letter. Step names must be unique within a workflow -- duplicate names cause a validation error. - -Step names matter because you reference step outputs in CEL expressions using `steps.STEP_NAME.output`. - -**Note on hyphenated step names in CEL:** When a step name contains hyphens (e.g., `fetch-data`), you can use dot notation in template strings (`{{ steps.fetch-data.output.body }}`), but in `if` expressions you must use bracket notation: `steps['fetch-data'].output.body`. This is because CEL interprets hyphens as subtraction in expression context. - -### Parallel Execution - -By default, Mantle builds a directed acyclic graph (DAG) from your steps and runs steps concurrently when their dependencies allow it. You control ordering with `depends_on` and through implicit dependencies detected from CEL expressions. - -**How dependencies are resolved:** - -- **Explicit dependencies** -- list step names in `depends_on` to declare that a step must wait for those steps to complete before it can start. -- **Implicit dependencies** -- Mantle analyzes CEL expressions in `params` and `if` fields. If a step references `steps.fetch-data.output`, the engine automatically adds `fetch-data` as a dependency. You do not need to list implicit dependencies in `depends_on`. -- **Skipped steps count as resolved** -- if a step is skipped (its `if` condition evaluated to `false`), downstream steps that depend on it are unblocked and can proceed. - -**Fan-out/fan-in example:** - -```yaml -name: fan-out-fan-in -description: Run two API calls in parallel, then merge results - -steps: - - name: fetch-users - action: http/request - params: - method: GET - url: https://api.example.com/users - - - name: fetch-orders - action: http/request - params: - method: GET - url: https://api.example.com/orders - - - name: merge-results - action: ai/completion - credential: openai - depends_on: - - fetch-users - - fetch-orders - params: - model: gpt-4o - prompt: > - Correlate these users and orders: - Users: {{ steps['fetch-users'].output.body }} - Orders: {{ steps['fetch-orders'].output.body }} -``` - -In this workflow, `fetch-users` and `fetch-orders` have no dependencies on each other, so they run concurrently. The `merge-results` step declares both as explicit dependencies via `depends_on` and waits for both to complete before it starts. - -## Retry Policy - -The retry policy controls what happens when a step fails. - -```yaml -retry: - max_attempts: 3 - backoff: exponential -``` - -| Field | Type | Required | Description | -|---|---|---|---| -| `max_attempts` | integer | Yes | Maximum number of attempts. Must be greater than 0. | -| `backoff` | string | No | Backoff strategy between retries. One of: `fixed`, `exponential`. | - -If `backoff` is omitted and `retry` is present, the default behavior depends on the engine implementation. - -## Timeout - -The `timeout` field accepts Go duration strings. These consist of a number followed by a unit suffix: - -| Unit | Suffix | Example | -|---|---|---| -| Milliseconds | `ms` | `500ms` | -| Seconds | `s` | `30s` | -| Minutes | `m` | `5m` | -| Hours | `h` | `1h` | - -You can combine units: `1m30s` means one minute and thirty seconds. - -The timeout must be a positive duration. `0s` and negative values are invalid. - -## CEL Expressions - -Mantle uses [CEL (Common Expression Language)](https://github.com/google/cel-go) for conditional logic and data access between steps. CEL expressions appear in two places: - -1. **`if` fields** -- determine whether a step runs -2. **Template strings in `params`** -- reference data from inputs and previous steps using `{{ expression }}` syntax - -### Available Variables - -| Variable | Description | -|---|---| -| `inputs.NAME` | Value of the input parameter `NAME`. | -| `steps.STEP_NAME.output` | Output of the step named `STEP_NAME`. The structure depends on the connector. | -| `env.NAME` | Value of the environment variable `NAME`. | -| `trigger.payload` | Request body from a webhook trigger, parsed as JSON. Only available for webhook-triggered executions. | - -### Expression Examples - -Reference an input: - -```yaml -url: "{{ inputs.url }}" -``` - -Reference a previous step's output: - -```yaml -prompt: "Summarize: {{ steps.fetch-data.output.body }}" -``` - -Conditional execution based on step output: - -```yaml -if: "steps.summarize.output.key_points.size() > 0" -``` - -Boolean logic: - -```yaml -if: "inputs.verbose == true && steps.fetch-data.output.status == 200" -``` - -String operations: - -```yaml -if: "steps.fetch-data.output.body.contains('error') == false" -``` - -### CEL Type Safety - -CEL is a strongly typed language. If you compare values of different types, the expression will fail at evaluation time. For example, `inputs.count > "5"` fails because you are comparing a number to a string. - -## Connectors - -Connectors define the actions a step can perform. Actions use a `connector/action` naming convention. - -### http/request - -Makes an HTTP request. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `method` | string | Yes | HTTP method: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`. | -| `url` | string | Yes | Request URL. | -| `headers` | map | No | HTTP headers as key-value pairs. | -| `body` | any | No | Request body. Objects are JSON-encoded. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `status` | number | HTTP response status code. | -| `headers` | map | Response headers. | -| `body` | string | Raw response body as a string. | -| `json` | any | Parsed response body. Only present when the response is valid JSON. | - -**Example:** - -```yaml -- name: create-item - action: http/request - params: - method: POST - url: https://api.example.com/items - headers: - Authorization: "Bearer {{ env.API_TOKEN }}" - Content-Type: application/json - body: - name: "New Item" - quantity: 5 -``` - -### ai/completion - -Sends a prompt to an OpenAI-compatible chat completion API and returns the result. Requires a credential with an API key -- see the [Secrets Guide](secrets-guide.md) for setup. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `provider` | string | No | AI provider to use: `openai` (default) or `bedrock`. | -| `model` | string | Yes | Model name (e.g., `gpt-4o`, `gpt-4o-mini`, `anthropic.claude-3-sonnet-20240229-v1:0`). | -| `prompt` | string | Yes | The user prompt to send. | -| `region` | string | No | AWS region for the Bedrock provider (e.g., `us-east-1`). Only used when `provider` is `bedrock`. | -| `system_prompt` | string | No | System message prepended to the conversation. | -| `output_schema` | object | No | JSON Schema for structured output. When set, the model returns JSON conforming to this schema. | -| `base_url` | string | No | Override the API base URL. Defaults to `https://api.openai.com/v1`. Use this for OpenAI-compatible providers like Azure, Ollama, or local models. | -| `tools` | list | No | Tool declarations for function calling. See [Tool Declarations](#tool-declarations). | -| `max_tool_rounds` | integer | No | Maximum number of LLM-tool interaction rounds. Default: `10` (from `engine.default_max_tool_rounds`). | -| `max_tool_calls_per_round` | integer | No | Maximum number of tool calls the LLM can make in a single round. Default: `10` (from `engine.default_max_tool_calls_per_round`). | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `text` | string | The raw completion text returned by the model. | -| `json` | any | If the response is valid JSON (e.g., from structured output), the parsed object. Only present when the response parses as JSON. | -| `tool_calls` | list | Tool invocations requested by the model. Each item has `id`, `type`, and `function` (with `name` and `arguments`). Only present when the model requests tool calls in the final response. | -| `finish_reason` | string | Why the model stopped generating. `stop` for normal text completion, `tool_calls` when the model requested tool invocations. | -| `model` | string | The model name as reported by the API. | -| `usage.prompt_tokens` | number | Number of tokens in the prompt. | -| `usage.completion_tokens` | number | Number of tokens in the completion. | -| `usage.total_tokens` | number | Total tokens used. | - -**Example -- basic completion:** - -```yaml -- name: summarize - action: ai/completion - credential: my-openai - params: - model: gpt-4o - prompt: "Summarize this in 3 bullet points: {{ steps.fetch-data.output.body }}" -``` - -**Example -- with system prompt and structured output:** - -```yaml -- name: extract-entities - action: ai/completion - credential: my-openai - timeout: 60s - params: - model: gpt-4o - system_prompt: "You are a data extraction assistant. Always respond with valid JSON." - prompt: "Extract all person names and companies from: {{ steps.fetch-data.output.body }}" - output_schema: - type: object - properties: - people: - type: array - items: - type: string - companies: - type: array - items: - type: string - required: - - people - - companies - additionalProperties: false -``` - -The structured output is available as `steps.extract-entities.output.json.people` and `steps.extract-entities.output.json.companies` in subsequent steps. - -**Example -- custom base URL (Ollama):** - -```yaml -- name: local-completion - action: ai/completion - params: - model: llama3 - base_url: http://localhost:11434/v1 - prompt: "Explain this error: {{ steps.fetch-logs.output.body }}" -``` - -**Example -- AWS Bedrock:** - -```yaml -- name: summarize - action: ai/completion - credential: aws-bedrock-creds - params: - provider: bedrock - model: anthropic.claude-3-sonnet-20240229-v1:0 - region: us-east-1 - prompt: "Summarize: {{ steps.fetch.output.body }}" -``` - -When running on AWS infrastructure with an IAM role attached (IRSA, instance profile, etc.), the `credential` field can be omitted -- the Bedrock provider uses the standard AWS credential chain automatically. - -**Authentication:** The AI connector reads the credential's `api_key` field (or `token` or `key` as fallbacks) and sends it as a Bearer token. If the credential includes an `org_id` field, it is sent as the `OpenAI-Organization` header. See the [Secrets Guide](secrets-guide.md) for how to create an `openai`-type credential. - -#### Tool Declarations - -Tools let the LLM call back into Mantle connectors during a completion. When you declare `tools` on an `ai/completion` step, the engine runs a multi-turn loop: it sends the prompt to the LLM, the LLM may request tool calls, the engine executes those calls using connector actions, feeds the results back to the LLM, and repeats until the LLM produces a final text response or the configured limits are reached. - -Each tool in the `tools` list has the following schema: - -| Field | Type | Required | Description | -|---|---|---|---| -| `name` | string | Yes | Tool name exposed to the LLM. | -| `description` | string | No | Human-readable description of what the tool does. Helps the LLM decide when to use it. | -| `input_schema` | object | No | JSON Schema describing the tool's input parameters. | -| `action` | string | Yes | Connector action to invoke when the LLM calls this tool (e.g., `http/request`, `postgres/query`). | -| `params` | map | No | Static parameters merged with the LLM-provided arguments when the tool is invoked. | - -**Example -- tool use with web search:** - -```yaml -- name: research-assistant - action: ai/completion - credential: openai - params: - model: gpt-4o - prompt: "Find the current population of Seattle and summarize the top 3 industries." - max_tool_rounds: 5 - tools: - - name: web_search - description: "Search the web for current information" - input_schema: - type: object - properties: - query: - type: string - description: "Search query" - required: - - query - action: http/request - params: - method: GET - url: "https://api.search.example.com/search" -``` - -The LLM sees `web_search` as an available function. When it decides to call `web_search(query="Seattle population 2026")`, the engine executes the `http/request` action with the merged parameters and returns the result to the LLM. This continues for up to `max_tool_rounds` rounds. - -If the LLM exhausts all rounds without producing a final text response, the engine makes one last call asking the LLM to summarize with the information gathered so far. - -### slack/send - -Sends a message to a Slack channel via the [chat.postMessage](https://api.slack.com/methods/chat.postMessage) API. Requires a credential with a Slack Bot User OAuth Token. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `channel` | string | Yes | Slack channel ID (e.g., `C01234ABCDE`). Use the channel ID, not the channel name. | -| `text` | string | Yes | Message text. Supports Slack [mrkdwn](https://api.slack.com/reference/surfaces/formatting) formatting. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `ok` | boolean | `true` if the message was sent successfully. | -| `ts` | string | Slack message timestamp. Use this to reference the message in follow-up API calls. | -| `channel` | string | The channel ID where the message was posted. | - -**Example:** - -```yaml -- name: notify-team - action: slack/send - credential: slack-bot - params: - channel: "C01234ABCDE" - text: "Deployment complete: {{ steps.deploy.output.body }}" -``` - -**Authentication:** The Slack connector reads the credential's `token` field and sends it as a Bearer token. Create a credential of type `bearer` with a `token` field containing your Slack Bot User OAuth Token: - -```bash -mantle secrets create --name slack-bot --type bearer --field token=xoxb-your-bot-token -``` - -### slack/history - -Reads recent messages from a Slack channel via the [conversations.history](https://api.slack.com/methods/conversations.history) API. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `channel` | string | Yes | Slack channel ID (e.g., `C01234ABCDE`). | -| `limit` | number | No | Maximum number of messages to return. Default: `10`. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `ok` | boolean | `true` if the request was successful. | -| `messages` | list | Array of message objects. Each message contains fields like `text`, `user`, `ts`, and `type`. | - -**Example:** - -```yaml -- name: read-channel - action: slack/history - credential: slack-bot - params: - channel: "C01234ABCDE" - limit: 5 - -- name: summarize-messages - action: ai/completion - credential: my-openai - params: - model: gpt-4o - prompt: "Summarize these Slack messages: {{ steps['read-channel'].output.messages }}" -``` - -### postgres/query - -Executes a parameterized SQL query against an external Postgres database. The connector opens a connection per step execution and closes it afterward. Supports both read queries (`SELECT`, `WITH`) and write statements (`INSERT`, `UPDATE`, `DELETE`). - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `query` | string | Yes | SQL query to execute. Use `$1`, `$2`, etc. for parameterized values. | -| `args` | list | No | Ordered list of values to substitute into the parameterized query. | - -**Output (SELECT/WITH queries):** - -| Field | Type | Description | -|---|---|---| -| `rows` | list | Array of row objects, each mapping column names to values. Empty array if no rows match. | -| `row_count` | number | Number of rows returned. | - -**Output (INSERT/UPDATE/DELETE statements):** - -| Field | Type | Description | -|---|---|---| -| `rows_affected` | number | Number of rows affected by the statement. | - -**Example -- read query:** - -```yaml -- name: fetch-users - action: postgres/query - credential: my-database - params: - query: "SELECT id, email FROM users WHERE active = $1 LIMIT $2" - args: - - true - - 100 -``` - -**Example -- write statement:** - -```yaml -- name: update-status - action: postgres/query - credential: my-database - params: - query: "UPDATE orders SET status = $1 WHERE id = $2" - args: - - "shipped" - - "{{ steps['create-order'].output.json.order_id }}" -``` - -**Authentication:** The Postgres connector reads the database connection URL from the credential's `url` field (or `key` as a fallback). Create a credential with the full Postgres connection string: - -```bash -mantle secrets create --name my-database --type generic --field url=postgres://user:pass@host:5432/dbname?sslmode=require -``` - -### email/send - -Sends an email via SMTP. Supports plaintext and HTML content. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `to` | string or list | Yes | Recipient email address(es). A single string or a list of strings. | -| `from` | string | Yes | Sender email address. | -| `subject` | string | Yes | Email subject line. | -| `body` | string | Yes | Email body content. | -| `html` | boolean | No | Set to `true` to send the body as HTML. Default: `false` (plaintext). | -| `smtp_host` | string | No | SMTP server hostname. Can also be provided via credential. | -| `smtp_port` | string | No | SMTP server port. Default: `587`. Can also be provided via credential. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `sent` | boolean | `true` if the email was sent successfully. | -| `to` | string | Comma-separated list of recipient addresses. | -| `subject` | string | The subject line that was sent. | - -**Example:** - -```yaml -- name: send-report - action: email/send - credential: smtp-creds - params: - to: - - "alice@example.com" - - "bob@example.com" - from: "reports@example.com" - subject: "Daily Report — {{ steps.generate.output.json.date }}" - body: "{{ steps.generate.output.json.html_report }}" - html: true -``` - -**Authentication:** The email connector reads `username`, `password`, `host`, and `port` from the credential. If `host` or `port` are not in the credential, they fall back to the `smtp_host` and `smtp_port` params. Create a `basic` credential with SMTP fields: - -```bash -mantle secrets create --name smtp-creds --type basic \ - --field username=apikey \ - --field password=SG.your-sendgrid-key \ - --field host=smtp.sendgrid.net \ - --field port=587 -``` - -### s3/put - -Uploads an object to an S3-compatible storage bucket. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `bucket` | string | Yes | S3 bucket name. | -| `key` | string | Yes | Object key (path) within the bucket. | -| `content` | string | Yes | Object content as a string. | -| `content_type` | string | No | MIME type for the object. Default: `application/octet-stream`. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `bucket` | string | The bucket the object was uploaded to. | -| `key` | string | The object key. | -| `size` | number | Size of the uploaded content in bytes. | - -**Example:** - -```yaml -- name: upload-report - action: s3/put - credential: aws-s3 - params: - bucket: "my-reports" - key: "reports/{{ steps.generate.output.json.date }}.json" - content: "{{ steps.generate.output.json.report }}" - content_type: "application/json" -``` - -### s3/get - -Downloads an object from an S3-compatible storage bucket. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `bucket` | string | Yes | S3 bucket name. | -| `key` | string | Yes | Object key (path) within the bucket. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `bucket` | string | The bucket the object was downloaded from. | -| `key` | string | The object key. | -| `content` | string | Object content as a string. | -| `size` | number | Size of the downloaded content in bytes. | -| `content_type` | string | MIME type of the object as reported by S3. | - -**Example:** - -```yaml -- name: download-config - action: s3/get - credential: aws-s3 - params: - bucket: "my-configs" - key: "app/config.json" -``` - -### s3/list - -Lists objects in an S3-compatible storage bucket, with optional prefix filtering. - -**Params:** - -| Param | Type | Required | Description | -|---|---|---|---| -| `bucket` | string | Yes | S3 bucket name. | -| `prefix` | string | No | Filter results to keys that start with this prefix. | - -**Output:** - -| Field | Type | Description | -|---|---|---| -| `bucket` | string | The bucket that was listed. | -| `objects` | list | Array of objects. Each object has `key` (string), `size` (number), and `last_modified` (string, RFC 3339). | - -**Example:** - -```yaml -- name: list-reports - action: s3/list - credential: aws-s3 - params: - bucket: "my-reports" - prefix: "reports/2026/" -``` - -**S3 Authentication:** All S3 connectors (`s3/put`, `s3/get`, `s3/list`) read the following fields from the credential: - -| Field | Required | Description | -|---|---|---| -| `access_key` | Yes | AWS access key ID. | -| `secret_key` | Yes | AWS secret access key. | -| `region` | No | AWS region. Default: `us-east-1`. | -| `endpoint` | No | Custom S3 endpoint URL. Use this for S3-compatible services like MinIO, DigitalOcean Spaces, or Backblaze B2. | - -Create a credential for S3: - -```bash -mantle secrets create --name aws-s3 --type generic \ - --field access_key=AKIA... \ - --field secret_key=wJalr... \ - --field region=us-west-2 -``` - -For S3-compatible services, add an `endpoint` field: - -```bash -mantle secrets create --name minio --type generic \ - --field access_key=minioadmin \ - --field secret_key=minioadmin \ - --field endpoint=http://localhost:9000 -``` - -## Triggers - -Triggers define how a workflow is started automatically when Mantle runs in server mode (`mantle serve`). A workflow can have zero, one, or multiple triggers. - -```yaml -triggers: - - type: cron - schedule: "*/5 * * * *" - - type: webhook - path: "/hooks/my-workflow" -``` - -Triggers are optional. Without them, the workflow can still be executed manually with `mantle run` or via the REST API (`POST /api/v1/run/{workflow}`). - -### Trigger Lifecycle - -Triggers are managed through the standard IaC lifecycle. When you run `mantle apply`: - -- **New triggers** in the YAML are registered with the server -- **Changed triggers** (e.g., updated cron schedule) are updated -- **Removed triggers** (deleted from the YAML) are deregistered - -You do not manage triggers separately. The workflow definition is the single source of truth. - -### Trigger Fields - -| Field | Type | Required | Description | -|---|---|---|---| -| `type` | string | Yes | Trigger type. One of: `cron`, `webhook`. | -| `schedule` | string | Cron only | Cron expression defining the schedule. Required when `type` is `cron`. | -| `path` | string | Webhook only | URL path for the webhook endpoint. Required when `type` is `webhook`. | - -### Cron Triggers - -Cron triggers execute the workflow on a recurring schedule. The `schedule` field uses standard 5-field cron syntax: - -``` -┌───────────── minute (0-59) -│ ┌───────────── hour (0-23) -│ │ ┌───────────── day of month (1-31) -│ │ │ ┌───────────── month (1-12) -│ │ │ │ ┌───────────── day of week (0-6, Sunday=0) -│ │ │ │ │ -* * * * * -``` - -**Supported syntax:** - -| Syntax | Meaning | Example | -|---|---|---| -| `*` | Every value | `* * * * *` -- every minute | -| `*/N` | Every N intervals | `*/5 * * * *` -- every 5 minutes | -| `N-M` | Range from N to M | `0 9-17 * * *` -- every hour from 9 AM to 5 PM | -| `N,M,O` | Comma-separated list | `0 0 1,15 * *` -- 1st and 15th of the month | - -**Examples:** - -```yaml -# Every 5 minutes -triggers: - - type: cron - schedule: "*/5 * * * *" - -# Daily at midnight -triggers: - - type: cron - schedule: "0 0 * * *" - -# Weekdays at 9 AM -triggers: - - type: cron - schedule: "0 9 * * 1-5" - -# Every hour on the hour -triggers: - - type: cron - schedule: "0 * * * *" -``` - -The cron scheduler polls every 30 seconds. Executions may start up to 30 seconds after the scheduled time. - -### Webhook Triggers - -Webhook triggers execute the workflow when an HTTP POST request is received at the configured path. The request body is available inside the workflow as `trigger.payload`. - -```yaml -triggers: - - type: webhook - path: "/hooks/deploy-notifier" -``` - -When the server receives a POST to `/hooks/deploy-notifier`, it starts a new execution. The full request body is parsed as JSON and made available through the `trigger.payload` variable in CEL expressions: - -```yaml -name: deploy-notifier -triggers: - - type: webhook - path: "/hooks/deploy-notifier" - -steps: - - name: notify - action: http/request - params: - method: POST - url: https://hooks.slack.com/services/T00/B00/xxx - body: - text: "Deployed {{ trigger.payload.repo }} to {{ trigger.payload.environment }}" -``` - -Triggering the webhook: - -```bash -curl -X POST http://localhost:8080/hooks/deploy-notifier \ - -H "Content-Type: application/json" \ - -d '{"repo": "my-app", "environment": "production"}' -``` - -### Multiple Triggers - -A workflow can have multiple triggers of different types. Each trigger independently starts a new execution: - -```yaml -triggers: - - type: cron - schedule: "0 * * * *" - - type: webhook - path: "/hooks/my-workflow" -``` - -This workflow runs every hour on the hour via cron, and can also be triggered on demand via a webhook POST. - -## Validation Rules Summary - -Mantle validates the following rules when you run `mantle validate` or `mantle apply`: - -| Rule | Error Message | -|---|---| -| Workflow name is required | `name is required` | -| Workflow name must be kebab-case | `name must match ^[a-z][a-z0-9-]*$` | -| At least one step is required | `at least one step is required` | -| Input names must be snake_case | `input name must match ^[a-z][a-z0-9_]*$` | -| Input types must be valid | `type must be one of: string, number, boolean` | -| Step names are required | `step name is required` | -| Step names must be kebab-case | `step name must match ^[a-z][a-z0-9-]*$` | -| Step names must be unique | `duplicate step name "NAME"` | -| Step actions are required | `step action is required` | -| Retry max_attempts must be > 0 | `max_attempts must be greater than 0` | -| Retry backoff must be valid | `backoff must be one of: fixed, exponential` | -| Timeout must be a valid duration | `invalid duration: ...` | -| Timeout must be positive | `timeout must be a positive duration` | -| Dependency cycle detected | `cycle detected in step dependencies` | -| `depends_on` references undefined step | `references undefined step "NAME"` | - -Validation errors include line and column numbers when available, formatted as: - -``` -workflow.yaml:3:1: error: step name must match ^[a-z][a-z0-9-]*$ (steps[0].name) -``` - -## Minimal Valid Workflow - -The smallest valid workflow contains a name and one step with an action: - -```yaml -name: hello -steps: - - name: greet - action: http/request - params: - method: GET - url: https://httpbin.org/get -``` diff --git a/go.mod b/go.mod index f6ee62a..098784d 100644 --- a/go.mod +++ b/go.mod @@ -142,6 +142,6 @@ require ( google.golang.org/genproto v0.0.0-20250603155806-513f23925822 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260217215200-42d3e9bedb6d // indirect - google.golang.org/grpc v1.79.1 // indirect + google.golang.org/grpc v1.79.3 // indirect google.golang.org/protobuf v1.36.11 // indirect ) diff --git a/go.sum b/go.sum index 4488ad1..a1b4c8c 100644 --- a/go.sum +++ b/go.sum @@ -350,8 +350,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 h1: google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:kSJwQxqmFXeo79zOmbrALdflXQeAYcUbgS7PbpMknCY= google.golang.org/genproto/googleapis/rpc v0.0.0-20260217215200-42d3e9bedb6d h1:t/LOSXPJ9R0B6fnZNyALBRfZBH0Uy0gT+uR+SJ6syqQ= google.golang.org/genproto/googleapis/rpc v0.0.0-20260217215200-42d3e9bedb6d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= -google.golang.org/grpc v1.79.1 h1:zGhSi45ODB9/p3VAawt9a+O/MULLl9dpizzNNpq7flY= -google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ= +google.golang.org/grpc v1.79.3 h1:sybAEdRIEtvcD68Gx7dmnwjZKlyfuc61Dyo9pGXXkKE= +google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/internal/cli/root.go b/internal/cli/root.go index 9c12f25..2b3cfe1 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -15,7 +15,7 @@ BYOK, IaC-first, enterprise-grade, source-available. validate → plan → apply → run -Full documentation: https://mantle.dev/docs`, +Full documentation: https://mantle.dvflw.co/docs`, SilenceUsage: true, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { cfg, err := config.Load(cmd) diff --git a/internal/config/config.go b/internal/config/config.go index ff207cb..b0fbffa 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -5,7 +5,11 @@ import ( "crypto/rand" "encoding/hex" "fmt" + "log" + "net" + "net/url" "os" + "strings" "time" "github.com/spf13/cobra" @@ -129,7 +133,7 @@ func Load(cmd *cobra.Command) (*Config, error) { v := viper.New() // Defaults - v.SetDefault("database.url", "postgres://mantle:mantle@localhost:5432/mantle?sslmode=require") + v.SetDefault("database.url", "postgres://mantle:mantle@localhost:5432/mantle?sslmode=prefer") v.SetDefault("database.max_open_conns", 25) v.SetDefault("database.max_idle_conns", 25) v.SetDefault("database.conn_max_lifetime", 5*time.Minute) @@ -231,6 +235,21 @@ func Load(cmd *cobra.Command) (*Config, error) { return nil, err } + // Warn if database URL uses sslmode=prefer on a non-loopback host. + if dbURL := cfg.Database.URL; dbURL != "" { + if parsed, err := url.Parse(dbURL); err == nil { + host := parsed.Hostname() + ip := net.ParseIP(host) + isLoopback := host != "" && (strings.EqualFold(host, "localhost") || (ip != nil && ip.IsLoopback())) + if !isLoopback { + q := parsed.Query() + if q.Get("sslmode") == "prefer" { + log.Printf("WARNING: database URL uses sslmode=prefer for non-loopback host %q; consider sslmode=require for production", host) + } + } + } + } + // Generate default NodeID if not set. // Format: hostname:pid:random8chars — the random suffix ensures uniqueness // across Kubernetes container restarts where PID 1 is common. diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 09dab38..616289b 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -29,7 +29,7 @@ func TestLoad_Defaults(t *testing.T) { t.Fatalf("Load() error = %v", err) } - if cfg.Database.URL != "postgres://mantle:mantle@localhost:5432/mantle?sslmode=require" { + if cfg.Database.URL != "postgres://mantle:mantle@localhost:5432/mantle?sslmode=prefer" { t.Errorf("Database.URL = %q, want default", cfg.Database.URL) } if cfg.API.Address != ":8080" { @@ -97,7 +97,7 @@ func TestLoad_ImplicitConfigMissing_UsesDefaults(t *testing.T) { t.Fatalf("Load() error = %v, want nil (silent fallback)", err) } - if cfg.Database.URL != "postgres://mantle:mantle@localhost:5432/mantle?sslmode=require" { + if cfg.Database.URL != "postgres://mantle:mantle@localhost:5432/mantle?sslmode=prefer" { t.Errorf("Database.URL = %q, want default", cfg.Database.URL) } } diff --git a/marketing/growth-strategy.md b/marketing/growth-strategy.md index 8404ce1..0c8cea0 100644 --- a/marketing/growth-strategy.md +++ b/marketing/growth-strategy.md @@ -16,7 +16,7 @@ Mantle's YAML format is its most underrated growth asset. Unlike Python-based to **Actions:** -- **Example gallery on the website.** Ship 17 example workflows (already in `examples/`) as a browsable gallery at `mantle.dev/examples`. Each example gets its own page with the YAML, a plain-English description, and a one-line `mantle apply` command. This is the single highest-leverage content investment because each example page becomes an SEO landing page. +- **Example gallery on the website.** Ship 17 example workflows (already in `examples/`) as a browsable gallery at `mantle.dvflw.co/examples`. Each example gets its own page with the YAML, a plain-English description, and a one-line `mantle apply` command. This is the single highest-leverage content investment because each example page becomes an SEO landing page. - **`mantle init --template` command.** Let users scaffold from templates: `mantle init --template=slack-summarizer`. Templates live in a public GitHub repo. Community members can submit templates via PR, creating a contribution flywheel. - **Workflow-of-the-week series.** Publish one practical workflow every week for the first 12 weeks on the blog and cross-post to dev.to, Hashnode, and r/devops. Each post walks through a real problem, shows the YAML, and ends with `mantle apply`. Topics: 1. Summarize Slack channels daily with GPT-4o @@ -262,14 +262,14 @@ Too early for a formal program at v0.1.0. Instead, identify and cultivate organi Create a simple badge that users can add to their README files: ```markdown -[![Built with Mantle](https://img.shields.io/badge/built%20with-Mantle-00ff88)](https://mantle.dev) +[![Built with Mantle](https://img.shields.io/badge/built%20with-Mantle-00ff88)](https://mantle.dvflw.co) ``` This serves two purposes: it signals adoption (social proof for the project) and it creates discovery events (anyone reading that README sees Mantle). Mention the badge in the "Getting Started" docs and in the post-quickstart success message. ### Example Workflow Gallery -Host at `mantle.dev/examples` with: +Host at `mantle.dvflw.co/examples` with: - All 17 built-in examples, categorized (AI, HTTP, Slack, Data, DevOps) - A "Community" section for user-submitted workflows - Each workflow page has: description, full YAML, required connectors/credentials, one-line apply command @@ -375,7 +375,7 @@ Sorted by expected impact relative to effort. Run 2-3 experiments concurrently. ### Experiment 4: GitHub Action for CI Validation - **Hypothesis:** A `mantle-action` GitHub Action will create organic visibility in adopters' CI pipelines and generate 50+ new discovery events per month. -- **Metric:** GitHub Action installs, referral traffic from GitHub to mantle.dev. +- **Metric:** GitHub Action installs, referral traffic from GitHub to mantle.dvflw.co. - **Effort:** Medium (4-6 hours to build and publish the action). - **Expected impact:** Medium. Slow burn but compounds over time. - **How to measure:** GitHub Marketplace install count, referral analytics. @@ -395,7 +395,7 @@ Sorted by expected impact relative to effort. Run 2-3 experiments concurrently. ### Experiment 6: Dev.to / Hashnode Cross-Posting - **Hypothesis:** Cross-posting technical blog content to dev.to and Hashnode will generate 2000+ additional impressions per post with near-zero marginal effort. -- **Metric:** Dev.to views, reactions, and referral traffic to mantle.dev. +- **Metric:** Dev.to views, reactions, and referral traffic to mantle.dvflw.co. - **Effort:** Low (30 minutes per post to cross-post with canonical URL). - **Expected impact:** Low-medium per post, but compounds over 12 weeks of content. - **How to measure:** Dev.to analytics, Hashnode analytics, Cloudflare referrals. diff --git a/marketing/launch-plan.md b/marketing/launch-plan.md index ccb997e..58bf9c7 100644 --- a/marketing/launch-plan.md +++ b/marketing/launch-plan.md @@ -460,7 +460,7 @@ Would love feedback on the design, the YAML format, the connector model, or anything else. GitHub: https://github.com/dvflw/mantle -Docs: https://mantle.dev/docs +Docs: https://mantle.dvflw.co/docs ``` ### Timing @@ -620,7 +620,7 @@ that fits your existing practices -- git, code review, CI/CD, IaC -- give Mantle a look. GitHub: github.com/dvflw/mantle -Docs: mantle.dev/docs +Docs: mantle.dvflw.co/docs Feedback, issues, and contributions are welcome. ``` @@ -1032,7 +1032,7 @@ The target audience respects technical depth and authenticity. Every post should Prepare these before launch day: - [ ] GitHub repo public with polished README, CHANGELOG, examples/, docs/ -- [ ] Marketing website live at mantle.dev +- [ ] Marketing website live at mantle.dvflw.co - [ ] Twitter/X account created and profile optimized - [ ] LinkedIn company page created - [ ] Dev.to account created diff --git a/marketing/website-brief.md b/marketing/website-brief.md index 751ecb6..70b5ec4 100644 --- a/marketing/website-brief.md +++ b/marketing/website-brief.md @@ -13,7 +13,7 @@ Generate a single-page marketing website for the v0.1.0 launch of Mantle, a head **Positioning:** Terraform for workflow automation, with native AI. **License:** BSL 1.1 (source-available). Do NOT call this "open source" anywhere on the page. Use "source-available" when referencing the license model. **GitHub:** https://github.com/dvflw/mantle -**Docs:** https://mantle.dev/docs (placeholder) +**Docs:** https://mantle.dvflw.co/docs (placeholder) ### Tone and Voice - Technical, confident, developer-first @@ -64,7 +64,7 @@ Single binary. Postgres state. Bring your own keys. **CTA buttons:** - Primary (green accent, solid): "View on GitHub" -> https://github.com/dvflw/mantle -- Secondary (outlined): "Get Started" -> https://mantle.dev/docs/getting-started +- Secondary (outlined): "Get Started" -> https://mantle.dvflw.co/docs/getting-started **Code block** (right side, in a faux terminal window with traffic-light dots and title bar showing "fetch-and-summarize.yaml"): @@ -362,7 +362,7 @@ mantle run hello-world execution, cron triggers, webhooks, and multi-turn tool use. ``` -**CTA button:** "Read the full Getting Started guide" -> https://mantle.dev/docs/getting-started +**CTA button:** "Read the full Getting Started guide" -> https://mantle.dvflw.co/docs/getting-started --- @@ -372,7 +372,7 @@ execution, cron triggers, webhooks, and multi-turn tool use. **Column 1: Product** - GitHub (https://github.com/dvflw/mantle) -- Documentation (https://mantle.dev/docs) +- Documentation (https://mantle.dvflw.co/docs) - Examples (https://github.com/dvflw/mantle/tree/main/examples) - Changelog (https://github.com/dvflw/mantle/blob/main/CHANGELOG.md) diff --git a/site/astro.config.mjs b/site/astro.config.mjs index eb6c1ae..e987e01 100644 --- a/site/astro.config.mjs +++ b/site/astro.config.mjs @@ -4,7 +4,7 @@ import mdx from '@astrojs/mdx'; import sitemap from '@astrojs/sitemap'; export default defineConfig({ - site: 'https://mantle.dev', + site: 'https://mantle.dvflw.co', integrations: [mdx(), sitemap()], vite: { plugins: [tailwindcss()], diff --git a/site/src/components/DocsSidebar.astro b/site/src/components/DocsSidebar.astro index b7cd7f7..c46dba7 100644 --- a/site/src/components/DocsSidebar.astro +++ b/site/src/components/DocsSidebar.astro @@ -93,7 +93,7 @@ function isParentActive(item: NavItem): boolean { -