[Feature] on_failure workflow-level error handler

[michaelmcnees]

Problem

Currently, Mantle supports continue_on_error: true on individual steps (shipped in v0.3.0), which allows downstream steps to check steps['name'].error and react. However, there is no workflow-level mechanism to guarantee that cleanup or notification steps always run when a workflow fails, succeeds, or finishes.

Design (from grill-me session, 2026-03-26)

Three Lifecycle Hooks

Hook	Fires when	Use case
on_success	Main workflow completes with no unhandled failures	Downstream triggers, success notifications
on_failure	Main workflow step fails (without continue_on_error) or workflow times out	Alerts, cleanup, incident creation
on_finish	Always — success, failure, or timeout (but NOT cancellation)	Resource cleanup, audit logging

YAML Schema

name: my-workflow
timeout: 5m
steps:
  - name: fetch-data
    action: http/request
    params:
      url: https://api.example.com/data

hooks:
  timeout: 2m    # optional aggregate cap for all hook execution
  on_success:
    - name: notify-team
      action: slack/send
      credential: slack-bot
      params:
        channel: "#ops"
        text: "Workflow completed successfully"
  on_failure:
    - name: alert-ops
      action: slack/send
      credential: slack-bot
      params:
        channel: "#ops-alerts"
        text: "Failed at {{ execution.failed_step }} ({{ execution.failed_in }}): {{ execution.error }}"
  on_finish:
    - name: cleanup
      action: http/request
      params:
        method: POST
        url: "https://api.example.com/cleanup"

Execution Order (Fixed)

1. Conditional hook runs first (on_success OR on_failure, never both from main workflow)
2. on_finish always runs last (except on cancellation)

Main workflow succeeds → on_success → on_finish
Main workflow fails    → on_failure → on_finish
Main workflow timeout  → on_failure → on_finish
Main workflow cancel   → no hooks

Hook Step Behavior

• Flat list, executed sequentially — no depends_on. For complex failure handling, call a child workflow instead.
• Full step feature support: credential, timeout, retry, if, continue_on_error, artifacts, registry_credential
• Names unique within their block, not globally. A hook step can share a name with a main step.
• Referenced as hooks['step-name'] in CEL expressions (separate namespace from steps['step-name'])

Hook Failure Behavior

Hooks are best-effort — they do NOT alter the workflow execution status.

This matches industry consensus (GitHub Actions, Argo Workflows, Airflow, n8n, Prefect all prevent hook failures from changing parent status or cascading).

• Hook step fails → halt remaining steps in that block → next block still runs
• continue_on_error: true on a hook step → record failure, continue to next step in block
• on_finish always runs regardless of on_success/on_failure outcome
• No cascading: on_success failure does NOT trigger on_failure
• Hook failures are logged, audited, and emitted as metrics

CEL Context in Hooks

Variable	Description
steps['name'].output	Main workflow step outputs
steps['name'].error	Main workflow step errors
hooks['name'].output	Hook step outputs (accumulates across blocks)
hooks['name'].error	Hook step errors
execution.status	"completed", "failed", or "timed_out"
execution.error	Error string or null
execution.failed_step	Name of the step that caused failure, or null
execution.failed_in	"steps", "on_success", "on_failure", "on_finish", or null
inputs, trigger, env, artifacts	Same as main workflow steps

Timeouts

• timeout (workflow-level) — covers main steps only
• hooks.timeout (optional) — separate aggregate cap for all hook execution
• Individual hook steps can have their own timeout

Cancellation

mantle cancel skips all hooks. Cancellation is intentional — the user doesn't need alerts about their own action.

Metrics

• mantle_hook_steps_total{workflow, hook, step, status} — counter per hook step execution
• mantle_hook_steps_failed_total{workflow, hook, step} — counter for hook failures

What This Does NOT Do

• Hooks do not cascade (on_success failure does not trigger on_failure)
• Hooks do not alter workflow execution status
• Hooks do not support depends_on (use child workflows for complex error handling)
• continue_on_error workflows that "complete" with step errors do NOT trigger on_failure — the execution status is completed

References

• v0.3.0 continue_on_error implementation
• Industry survey: GitHub Actions, Argo Workflows, Temporal, Airflow, n8n, Prefect — all prevent hook cascade and status mutation
• GitHub issue feat: add continue_on_error / on_failure step option for error notification workflows #29