[Feature] Workflow concurrency controls (max parallel executions)

[michaelmcnees]

Problem

There are no concurrency controls. A webhook flood can spawn unlimited parallel executions of the same workflow, overwhelming infrastructure. Fan-out steps can spawn unlimited parallel branches.

Design (from grill-me session, 2026-03-25)

Three Concurrency Dimensions

Dimension	Config Location	Default
Per-workflow max_parallel_executions	Workflow YAML	unlimited (0)
Per-team global limit	mantle.yaml default + per-team DB override	unlimited (0)
Per-step max_parallel	Step YAML (fan-out steps)	unlimited (0)

YAML Schema

name: my-workflow
max_parallel_executions: 5
on_limit: queue    # "queue" (default) or "reject"

steps:
  - name: process-batch
    action: http/request
    depends_on: [split-data]
    max_parallel: 3
    params: ...

mantle.yaml:

engine:
  max_concurrent_executions_per_team: 50  # default, overridable per-team in DB

New Execution Status

queued → pending → running → {completed | failed | cancelled}

• queued = waiting for concurrency slot
• cancelled works from both queued and running states

Enforcement Points

• Per-team limit: checked first at execution creation, advisory lock on team ID
• Per-workflow limit: checked second, advisory lock on hash(workflow_name)
• Both use pg_advisory_xact_lock inside a transaction that counts running executions and either inserts (pending) or queues (queued)
• Per-step max_parallel: enforced in ReadySteps in the orchestrator — counts running siblings from the same fan-out, only releases up to max_parallel

Queue Promotion

• Primary: completion-triggered — when an execution reaches a terminal state (completed/failed/cancelled), it checks for queued executions for the same workflow and promotes the oldest
• Backup: 30s poller scans for queued executions with available slots (catches crashes during promotion)

Webhook Behavior

• on_limit: queue (default) → 202 Accepted, execution created as queued
• on_limit: reject → 429 Too Many Requests, no execution created
• Docs recommend exponential backoff (no Retry-After header — dynamic workloads prevent reliable estimates)

CLI Behavior

• mantle run respects concurrency limits by default
• mantle run --force bypasses limits (operator override)

Metrics

• mantle_executions_queued (gauge) — current queue depth per workflow
• mantle_executions_rejected_total (counter) — rejected due to limit
• mantle_queue_wait_duration_seconds (histogram) — time in queue before promotion

Edge Cases

• Any terminal state (completed, failed, cancelled) frees a concurrency slot
• Orphaned running executions after crash count against limits temporarily — accepted tradeoff. Step reaper handles recovery, 30s backup poller covers queue promotion.
• 0 means unlimited for all three dimensions

Migration

• Add max_parallel_executions, on_limit columns to workflow_definitions (or store in YAML content — TBD at implementation)
• Add max_concurrent_executions column to teams table (nullable, overrides config default)
• Add max_parallel to step schema in workflow YAML
• No new tables needed — advisory locks are ephemeral