BEAM clustering: multi-node workflow distribution (v0.3.8)

Multi-node support via Erlang distribution. Each node runs its own
sequential executor independently; nodes exchange workflow outputs
via send_to_workflow and receive_from_workflow core skills.

New features:
- ClusterManager GenServer: auto-registration, node monitoring
  (nodeup/nodedown), remote workflow triggers via :rpc.call
- send_to_workflow skill: sends data to a workflow on another node
  (5s default timeout, configurable per-step)
- receive_from_workflow skill: gate that must be step 1 to accept
  remote triggers, optional allowed_nodes ACL
- Cluster admin UI page with node status and ping
- Workflow "Run on" dropdown for node assignment (cluster-wide or
  pinned to specific node)
- Cross-node config sync via PubSub over BEAM distribution
- Gateway node assignment: telegram.node/discord.node in Config
  with auto-assign on enable. Single-node ignores assignments.
- Global alert bar for unassigned gateways in cluster mode
- docker-compose_swarm.yml for multi-node local testing
- EPMD bundled in runtime image for long-name distribution
- Node name shown in nav bar across all pages
- Workflow cluster role badges (recv/send/bridge)

Infrastructure:
- 3 migrations: cluster_nodes table, node column on workflow_runs
  and workflows
- Default node name alexclaw@node1.local (long names everywhere)
- telegram.enabled config for explicit gateway control
- Config.Loader subscribes to PubSub for cross-node ETS sync
- Fixed 20 pre-existing ShellTest failures (missing shell.enabled)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: thatsme

.gitignore

@@ -32,8 +32,9 @@ alex_claw-*.tar
 /docs/*
 !/docs/screenshot/
 
-# Personal docker-compose variants
+# Personal docker-compose variants (except swarm which is part of clustering)
 docker-compose_*.yml
+!docker-compose_swarm.yml
 
 # Claude Code
 .claude/

ALEXCLAW_ARCHITECTURE.md

@@ -31,6 +31,7 @@ AlexClaw.Application (one_for_one)
   ├── Registry (CircuitBreakerRegistry)  # Process registry for per-skill circuit breakers
   ├── AlexClaw.Skills.CircuitBreakerSupervisor  # DynamicSupervisor — per-skill circuit breakers
   ├── AlexClaw.SkillSupervisor        # DynamicSupervisor — spawns skill worker processes
+  ├── AlexClaw.Cluster.Manager       # Node registration, discovery, remote workflow triggers
   ├── AlexClaw.Scheduler              # Quantum cron scheduler
   ├── AlexClaw.Workflows.SchedulerSync # Syncs DB workflow schedules into Quantum jobs
   ├── AlexClaw.Gateway.Telegram       # Telegram long-polling bot
@@ -283,6 +284,29 @@ The circuit breaker wraps skill execution transparently in `Executor.execute_ste
 
 When a workflow contains a `telegram_notify` step, the executor sends a start notification when the workflow begins and a failure notification if any step fails before reaching the notify step.
 
+### Multi-Node Distribution — `AlexClaw.Cluster.Manager`
+
+AlexClaw supports multi-node BEAM clustering. Each node runs its own sequential executor — no parallel step changes. Nodes exchange workflow outputs over Erlang distribution.
+
+**ClusterManager** is a GenServer that:
+- Auto-registers itself and connecting nodes via `:net_kernel.monitor_nodes/1`
+- Pings known nodes from the DB 5 seconds after boot
+- Updates node status on `:nodeup`/`:nodedown` events
+- Handles incoming remote workflow triggers via `receive_workflow_data/3` (called by `:rpc.call` from remote nodes)
+- Validates the target workflow has `receive_from_workflow` as step 1 before allowing execution
+
+**Cross-node workflow flow:**
+1. Node A runs a workflow with a `send_to_workflow` step
+2. `send_to_workflow` calls `:rpc.call(node_b, ClusterManager, :receive_workflow_data, ...)` with a 5s default timeout
+3. Node B's ClusterManager validates the gate, spawns the target workflow via `Task.Supervisor`
+4. The target workflow's `receive_from_workflow` step receives the data as input with `_source_node` in config
+
+**Node assignment:** Workflows have an optional `node` field. If set, only that node's scheduler picks it up. If null, it's cluster-wide (any node can run it). The admin UI shows a "Run on" dropdown populated from connected cluster nodes.
+
+**Gateway behavior in cluster:** Single-node mode ignores `telegram.node` and `discord.node` — gateways always start. In a cluster (connected peers detected), gateways only start on the assigned node. Setting `telegram.enabled = true` from a node's Config UI auto-assigns `telegram.node` to that node. Cross-node config changes propagate via PubSub over BEAM distribution.
+
+**Configuration:** `NODE_NAME` (default: `alexclaw@node1.local`) and `CLUSTER_COOKIE` env vars. Long names (containing `.`) use EPMD for distribution. The naming convention is `alexclaw@nodeN.local` — same format for single-node and swarm.
+
 ---
 
 ## Skills
@@ -306,6 +330,8 @@ Skills are Elixir modules implementing the `AlexClaw.Skill` behaviour (`run/1`,
 | `shell` | `Shell` | — | Execute whitelisted OS commands for container introspection |
 | `coder` | `Coder` | local | Autonomous skill generation from natural language goals |
 | `web_automation` | `WebAutomation` | — | Browser recording and headless replay via sidecar |
+| `send_to_workflow` | `SendToWorkflow` | — | Send data to a workflow on another BEAM node via RPC |
+| `receive_from_workflow` | `ReceiveFromWorkflow` | — | Gate: accepts remote triggers when placed as step 1 |
 
 ### Skill API — `AlexClaw.Skills.SkillAPI`
 

Dockerfile

@@ -51,6 +51,9 @@ RUN apk add --no-cache libstdc++ openssl ncurses-libs postgresql-client git
 WORKDIR /app
 
 COPY --from=build /app/_build/prod/rel/alex_claw ./
+
+# EPMD is needed for BEAM long-name distribution (clustering)
+COPY --from=build /usr/local/lib/erlang/erts-*/bin/epmd /usr/local/bin/epmd
 COPY entrypoint.sh ./
 RUN sed -i 's/\r$//' entrypoint.sh && chmod +x entrypoint.sh
 

INSTALLATION.md

@@ -51,6 +51,10 @@ ANTHROPIC_API_KEY=
 # GOOGLE_OAUTH_CLIENT_SECRET=
 # GOOGLE_OAUTH_REFRESH_TOKEN=
 
+# === Clustering (optional — multi-node) ===
+# NODE_NAME=alexclaw@node1.local
+# CLUSTER_COOKIE=generate_a_random_secret
+
 # === Advanced ===
 # ADMIN_PORT=5001
 ```
@@ -494,6 +498,68 @@ docker compose exec alexclaw bin/alex_claw rpc \
 
 ---
 
+## Clustering (Optional — Multi-Node)
+
+AlexClaw supports running multiple instances connected via BEAM distribution. Each node runs independently with its own gateway connections and scheduler, sharing a single PostgreSQL database.
+
+There are two Docker Compose files:
+
+| File | Purpose |
+|---|---|
+| `docker-compose.yml` | **Single node** (default). One AlexClaw instance + DB + web-automator |
+| `docker-compose_swarm.yml` | **Multi-node**. Two AlexClaw nodes + shared DB. Each node has its own port and node name |
+
+### Single Node (default)
+
+No extra config needed. Just `docker compose up -d` as described in Setup.
+
+The default node name is `alexclaw@node1.local`. Override via `NODE_NAME` in `.env` if needed. Single-node mode ignores gateway node assignments — Telegram and Discord always start.
+
+### Multi-Node (same machine)
+
+```bash
+docker compose -f docker-compose_swarm.yml up --build -d
+```
+
+This starts two nodes:
+- **node1** — `alexclaw@node1.local` on `localhost:5001`
+- **node2** — `alexclaw@node2.local` on `localhost:5002`
+
+Both share the same database and auto-discover each other on boot. The Cluster admin page (Admin > Cluster) shows connected nodes.
+
+To add more nodes, duplicate a node block in `docker-compose_swarm.yml` with a new hostname and port.
+
+### Cross-Node Workflows
+
+1. Create a workflow with `receive_from_workflow` as step 1 — this is the receiver
+2. Create a workflow with `send_to_workflow` as a step — configure `target_node` and `target_workflow`
+3. Run the sender workflow — data flows from one node to the other over BEAM distribution
+
+### Node Assignment
+
+Both workflows and gateways support node assignment — cluster-wide or pinned to a specific node. The database is always the source of truth.
+
+**Workflows:** each workflow has a "Run on" dropdown (visible when clustering is active):
+- **Cluster-wide** (default) — any node can run the scheduled workflow
+- **Specific node** — only that node's scheduler picks it up
+
+**Gateways (Telegram, Discord):** in Admin > Config, set `telegram.node` or `discord.node` to a node name. Only that node will connect the bot. Leave empty for cluster-wide (single-node default). This prevents multiple nodes from polling the same bot token — which causes API conflicts.
+
+### Environment Variables
+
+| Variable | Description | Example |
+|---|---|---|
+| `NODE_NAME` | BEAM node name. Use `.` for long names (recommended) | `alexclaw@node1.local` |
+| `CLUSTER_COOKIE` | Shared secret for inter-node authentication | `your_random_secret` |
+
+### Security
+
+- `CLUSTER_COOKIE` is the authentication mechanism between nodes — treat it like `SECRET_KEY_BASE`
+- EPMD (port 4369) must be reachable between nodes but should NOT be exposed publicly
+- Use firewall rules or private networks to restrict inter-node traffic
+
+---
+
 ## Running on a VPS / Cloud Server
 
 AlexClaw is designed to run on a local machine, but works fine on a VPS too. Telegram polling is outbound-only, so no inbound ports are needed for the bot itself.

README.md

@@ -26,6 +26,7 @@ AlexClaw monitors the world (RSS feeds, web sources, GitHub repositories, APIs),
 - **Persistent Memory with Semantic Search** — PostgreSQL + pgvector for knowledge storage. Deduplication by URL. Hybrid search combines vector cosine similarity and keyword matching — vector results are prioritized, keyword results fill gaps for exact matches. Embeddings are generated asynchronously via the LLM router (Gemini `gemini-embedding-001`, Ollama `nomic-embed-text`, or any OpenAI-compatible endpoint). 768-dimension vectors with HNSW index. All skills that store knowledge auto-embed in the background.
 - **Knowledge Base RAG** — Separate `knowledge_entries` table for documentation and reference material, isolated from news/conversation memory. Scraper skills fetch, chunk, and embed documentation from hexdocs.pm, Erlang/OTP source (GitHub), Elixir stdlib source, Learn You Some Erlang, and existing skill code. Chat integrates both Knowledge and Memory search with a context source selector. ~7200 embeddings across 6 knowledge kinds.
 - **Cron Scheduler** — Quantum-based. Jobs defined in config or DB.
+- **Multi-Node BEAM Clustering** — Multiple AlexClaw instances connected via Erlang distribution. Each node runs its own executor independently; nodes exchange workflow outputs via `send_to_workflow` and `receive_from_workflow` skills. Auto-discovery, node status monitoring, and per-workflow node assignment from the admin UI. `docker-compose_swarm.yml` included for local multi-node testing.
 
 ### Skills
 
@@ -48,6 +49,8 @@ AlexClaw monitors the world (RSS feeds, web sources, GitHub repositories, APIs),
 | `shell` | Execute whitelisted OS commands for container introspection (2FA-gated) |
 | `web_automation` | Browser automation via headless Playwright sidecar (**experimental**) |
 | `coder` | Generate dynamic skills from natural language via local LLM |
+| `send_to_workflow` | Send data to a workflow on another BEAM node |
+| `receive_from_workflow` | Gate: accepts remote triggers when placed as step 1 |
 | `hexdocs_scraper` | Scrape hexdocs.pm docs into knowledge base embeddings (dynamic) |
 | `erlang_docs_scraper` | Fetch Erlang/OTP docs from GitHub into knowledge base (dynamic) |
 | `lyse_scraper` | Scrape Learn You Some Erlang chapters into knowledge base (dynamic) |

ROADMAP.md

@@ -59,6 +59,10 @@ Execute OS commands inside the AlexClaw container via Telegram/Discord. 5-layer
 
 Local LLM generates dynamic skills from natural language goals via `/coder <goal>`. SkillAPI extended with `:skill_write`, `:skill_manage`, `:workflow_manage` permissions. Retry loop with error feedback, knowledge base RAG context, optional workflow creation. Zero cloud API cost (always uses `tier: :local`). Generated code passes full validation pipeline (namespace, behaviour, permissions). See [SELF_AWARENESS.md](docs/SELF_AWARENESS.md).
 
+### ~~Multi-Node BEAM Clustering~~ ✅ Completed (v0.3.8)
+
+Multiple AlexClaw instances connected via Erlang distribution exchange workflow outputs over BEAM. Each node runs its own sequential executor — no parallel step changes. ClusterManager GenServer handles auto-registration on connect, node monitoring (`:nodeup`/`:nodedown`), and remote workflow triggers via `:rpc.call`. Two new core skills: `send_to_workflow` (sends data to a workflow on another node, 5s default timeout) and `receive_from_workflow` (gate skill — must be step 1 to accept remote triggers, optional `allowed_nodes` ACL). Cluster admin UI page with node status and ping. Workflow "Run on" dropdown for node assignment (cluster-wide or pinned). `docker-compose_swarm.yml` for multi-node testing with long-name distribution (`alexclaw@nodeN.local`). EPMD bundled in runtime image.
+
 ---
 
 ## Someday

SECURITY.md

@@ -52,6 +52,23 @@ Set `github.webhook_secret` in Admin > Config (GitHub category).
 
 ---
 
+## Inter-Node Authentication (Clustering)
+
+Multi-node clusters authenticate via BEAM's distributed Erlang protocol:
+
+- All nodes must share the same `CLUSTER_COOKIE` (set via environment variable)
+- EPMD (Erlang Port Mapper Daemon) on port 4369 coordinates node discovery
+- Nodes without the correct cookie cannot join the cluster or trigger remote workflows
+- The `receive_from_workflow` gate skill provides an additional per-workflow access control layer via optional `allowed_nodes` config
+
+**Hardening recommendations:**
+- Generate `CLUSTER_COOKIE` with `openssl rand -base64 32` — treat it like `SECRET_KEY_BASE`
+- EPMD port (4369) and BEAM distribution ports (dynamic, high range) should NOT be exposed to the internet
+- Restrict inter-node traffic to private networks, VPCs, or Docker networks
+- When running across machines, use VPN or SSH tunnels between Docker hosts
+
+---
+
 ## Shell Skill (Container Introspection)
 
 The `/shell` command allows the owner to run OS commands inside the container

config/runtime.exs

@@ -30,4 +30,6 @@ if config_env() == :prod do
     ollama_enabled: System.get_env("OLLAMA_ENABLED") == "true",
     ollama_host: System.get_env("OLLAMA_HOST", "http://localhost:11434"),
     ollama_model: System.get_env("OLLAMA_MODEL", "llama3.2")
+
+  config :alex_claw, :node_name, System.get_env("NODE_NAME")
 end

docker-compose.yml

@@ -19,6 +19,7 @@ services:
       context: .
       args:
         BUILD_NUMBER: ${BUILD_NUMBER:-0}
+    hostname: node1.local
     restart: unless-stopped
     depends_on:
       db:
@@ -57,6 +58,10 @@ services:
       WEB_AUTOMATOR_ENABLED: ${WEB_AUTOMATOR_ENABLED:-false}
       WEB_AUTOMATOR_HOST: ${WEB_AUTOMATOR_HOST:-http://web-automator:6900}
 
+      # Clustering
+      NODE_NAME: ${NODE_NAME:-alexclaw@node1.local}
+      CLUSTER_COOKIE: ${CLUSTER_COOKIE:-alexclaw_default}
+
       # Dynamic skills
       SKILLS_DIR: /app/skills
     volumes:

docker-compose_swarm.yml

@@ -0,0 +1,91 @@
+services:
+  db:
+    image: pgvector/pgvector:pg17
+    restart: unless-stopped
+    environment:
+      POSTGRES_USER: ${DATABASE_USERNAME:-alexclaw}
+      POSTGRES_PASSWORD: ${DATABASE_PASSWORD}
+      POSTGRES_DB: alex_claw_prod
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U ${DATABASE_USERNAME:-alexclaw}"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  node1:
+    build:
+      context: .
+      args:
+        BUILD_NUMBER: ${BUILD_NUMBER:-0}
+    hostname: node1.local
+    restart: unless-stopped
+    depends_on:
+      db:
+        condition: service_healthy
+    ports:
+      - "5001:5001"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    environment:
+      DATABASE_USERNAME: ${DATABASE_USERNAME:-alexclaw}
+      DATABASE_PASSWORD: ${DATABASE_PASSWORD}
+      DATABASE_HOSTNAME: db
+      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
+      ADMIN_PASSWORD: ${ADMIN_PASSWORD}
+      TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
+      TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
+      GEMINI_API_KEY: ${GEMINI_API_KEY:-}
+      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+      OLLAMA_ENABLED: ${OLLAMA_ENABLED:-false}
+      OLLAMA_HOST: ${OLLAMA_HOST:-http://host.docker.internal:11434}
+      OLLAMA_MODEL: ${OLLAMA_MODEL:-}
+      LMSTUDIO_ENABLED: ${LMSTUDIO_ENABLED:-false}
+      LMSTUDIO_HOST: ${LMSTUDIO_HOST:-http://host.docker.internal:1234}
+      LMSTUDIO_MODEL: ${LMSTUDIO_MODEL:-}
+      NODE_NAME: alexclaw@node1.local
+      CLUSTER_COOKIE: alexclaw_swarm
+      SKILLS_DIR: /app/skills
+    volumes:
+      - skills_data:/app/skills
+
+  node2:
+    build:
+      context: .
+      args:
+        BUILD_NUMBER: ${BUILD_NUMBER:-0}
+    hostname: node2.local
+    restart: unless-stopped
+    depends_on:
+      db:
+        condition: service_healthy
+    ports:
+      - "5002:5001"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    environment:
+      DATABASE_USERNAME: ${DATABASE_USERNAME:-alexclaw}
+      DATABASE_PASSWORD: ${DATABASE_PASSWORD}
+      DATABASE_HOSTNAME: db
+      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
+      ADMIN_PASSWORD: ${ADMIN_PASSWORD}
+      TELEGRAM_BOT_TOKEN: ${TELEGRAM_BOT_TOKEN}
+      TELEGRAM_CHAT_ID: ${TELEGRAM_CHAT_ID}
+      GEMINI_API_KEY: ${GEMINI_API_KEY:-}
+      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+      OLLAMA_ENABLED: ${OLLAMA_ENABLED:-false}
+      OLLAMA_HOST: ${OLLAMA_HOST:-http://host.docker.internal:11434}
+      OLLAMA_MODEL: ${OLLAMA_MODEL:-}
+      LMSTUDIO_ENABLED: ${LMSTUDIO_ENABLED:-false}
+      LMSTUDIO_HOST: ${LMSTUDIO_HOST:-http://host.docker.internal:1234}
+      LMSTUDIO_MODEL: ${LMSTUDIO_MODEL:-}
+      NODE_NAME: alexclaw@node2.local
+      CLUSTER_COOKIE: alexclaw_swarm
+      SKILLS_DIR: /app/skills
+    volumes:
+      - skills_data:/app/skills
+
+volumes:
+  pgdata:
+  skills_data: