This is the canonical description of how FlowOS discovers, validates, isolates, runs, and observes plugins. It describes the code as it is; where an older document disagrees, this one is correct.
A plugin is an ordinary installed Python distribution that publishes two things through packaging metadata:
# The capability: what this plugin is, and which node types it owns.
[project.entry-points."flowforge.capabilities"]
"flowos.demo" = "flowos_demo_nodes:CAPABILITY"
# The implementations those node types resolve to.
[project.entry-points."flowforge.nodes"]
"flowos.demo.sum" = "flowos_demo_nodes.nodes:SumNode"
"flowos.demo.uppercase" = "flowos_demo_nodes.nodes:UppercaseNode"flowforge.capabilities and flowforge.nodes are entry-point group names,
not importable modules. Nothing imports them; plugins declare them.
The capability itself is built from the node manifests the plugin already wrote, so it cannot promise a node type the package does not implement:
from flowforge_sdk import capability_manifest
from flowos_demo_nodes.nodes import SumNode, UppercaseNode
CAPABILITY = capability_manifest(
capability_id="flowos.demo",
display_name="FlowOS Demo",
version="1.0.0",
nodes=[SumNode.manifest, UppercaseNode.manifest],
)examples/reference_plugin is the complete worked example, with tests.
Every node type must start with "<namespace>.", and the capability id must
start with the namespace. Node types are globally unique across the whole
deployment; two plugins claiming one node type is a startup failure, not a
race (see §5).
There are two discovery mechanisms and exactly one execution authority.
flowforge.nodes only |
flowforge.capabilities (+ nodes) |
|
|---|---|---|
| Discovered by | EntryPointNodeRegistry |
EntryPointPluginDiscovery |
| Bound | at composition time | during the lifespan, after validation |
| Isolated | no | yes, per capability |
| Credentials | from the node manifest | from the capability contract |
| Lifecycle | none | discovered → initializing → ready → stopping → stopped/failed |
| Intended for | built-in core nodes, small in-tree packages | every real integration |
Both end up in the same place: CapabilityRegistry is the single execution
authority. GET /nodes, workflow validation, the scheduler, the executor,
and credential resolution all resolve through it, so they cannot disagree
about what a node is.
The two mechanisms cannot collide, because the composition root computes the set of capability-publishing distributions from packaging metadata (no imports) and excludes them from the simple entry-point path:
capability_owned = capability_distributions()
entry_point_registry = EntryPointNodeRegistry(exclude_distributions=capability_owned)
bind_entry_point_nodes(capability_registry, entry_point_registry)A capability plugin's node types therefore become executable only once its plugin has been discovered, validated, started, and bound — never through the unisolated path as a side effect of being installed. If it fails to start, its nodes are unavailable, which is the correct fail-closed outcome.
bind_entry_point_nodes raises NodeBindingError rather than skipping a node
it cannot bind: a deployment that does not provide what its installed packages
claim should fail loudly at startup, not confusingly at execution time.
Process isolation in FlowOS means a process serves one capability of one distribution — not merely that a subprocess exists.
flowforge.infrastructure.plugins.isolation.load_capability_runtime(distribution, capability_id)
is the single mechanism, used by both hosting modes:
- Resolve the named installed distribution. Only its entry points are read, so no unrelated plugin is imported, registered, or reachable.
- Load its
flowforge.capabilitiesentry points and select the requested capability id. - Load its
flowforge.nodesentry points and keep only the node types the capability declares. An implemented-but-undeclared node is not served. - Reconcile: a declared node type with no implementation raises
PluginIsolationErrorat load time, so the divergence is a bounded start failure rather than a runtime surprise.
The result is a CapabilityNodeRegistry that can only ever resolve that one
capability's node types. Anything else answers CAPABILITY_NOT_FOUND.
ProcessPluginHost spawns:
python -m flowforge.infrastructure.plugins.process_plugin \
--distribution <installed-distribution> --capability <capability-id>
Both arguments are required — a runtime that had to guess what to serve is the
exact isolation failure this design exists to prevent. On a load failure the
process exits 2 and the host's start fails, bounded.
The host then verifies rather than trusts. initialize reconciles the
manifest the process reports against the manifest discovery found
(reconcile_manifest) and rejects the handshake on any disagreement about
capability id, version, runtime kind, or node set — in either direction. A
process cannot quietly serve a wider (or narrower) set of nodes than the host
validated and bound.
InProcessPluginHost applies the identical rule: each started plugin gets its
own capability-scoped registry, so one in-process plugin cannot execute
another's nodes. It shares an interpreter with the host, so it is a
correctness boundary, not a security one.
A plugin process runs with the same operating-system privileges as the host user. The process boundary bounds node dispatch, not the operating system. Treat plugin packages as untrusted code and confine them at the deployment layer — container, dedicated user, seccomp — when you do not control the package.
Settings.plugin_directories (FLOWOS_PLUGIN_DIRECTORIES, os.pathsep-separated):
- empty (default) — no directory restriction. Every installed distribution publishing a capability is discovered. The installed environment is the trust boundary; this is what an operator expects after installing a plugin.
- set — only distributions installed under those paths are discovered. Use this when third-party plugins live in a separately managed tree.
A broken plugin (unloadable entry point, non-manifest value, failed validation) is skipped with a warning; one bad plugin never hides the healthy ones. Two distributions publishing the same capability id: the first wins and the second is skipped with a warning.
PluginLifecycleService owns the state machine
(flowforge.domain.plugin.PluginState):
DISCOVERED ──► INITIALIZING ──► READY ⇄ RUNNING ──► STOPPING ──► STOPPED
│ ▲ │ │ │
│ │ └────────┴──► BACKOFF ──► STOPPED
│ └──────── restart ────────────┘
└───────────────────────────────────────────────────────► FAILED
Only transitions in the table are legal; terminal states have no outgoing
edges; FAILED always carries a reason.
BACKOFF is deliberately distinct from FAILED. FAILED means the runtime
has given up; BACKOFF means it has not. An operator needs to tell "this heals
itself in a moment" from "this needs me". A DEGRADED state was considered and
rejected: it would describe the same situation as BACKOFF plus a restart
count, which restart_count already carries.
PluginLifecycleService is also the supervisor. Keeping the two together is
deliberate — a separate supervisor would mutate the same lifecycle state
through a second channel, and the two owners would race. There is one owner,
and every transition goes through a per-plugin asyncio.Lock.
That lock is what makes the interesting races safe:
- a crash arriving during shutdown waits, then sees a terminal state and does nothing;
- a restart cannot begin until the previous instance is fully stopped, so a plugin never has two live sessions;
- a shutdown during backoff cancels the pending restart rather than racing it;
- a restart that starts but cannot be re-bound is stopped again, not leaked;
- a duplicate crash report does not double-spawn the plugin;
- a crash during startup consumes the restart budget exactly as a later crash does, so a plugin that dies before its handshake is not retried forever.
RestartPolicy governs the behaviour: bounded exponential backoff with jitter,
a budget counted in consecutive failures, and a stable-uptime window that
clears the streak. See docs/failure-semantics.md section 6.
- Readiness requires a complete handshake. A handshake missing its
capabilities or manifest is
FAILED, notREADY— the absence of an error is not evidence of success. - A plugin that starts but cannot be bound is shut down again, so a rejected plugin never leaves a live process or an unreachable session.
- Binding is atomic.
CapabilityRegistry.bindstages every runner and mutates only when all succeed, so a malformed manifest cannot half-register. - Crash detection.
ProcessPluginHostwatches both the process and the transport. An unexpected exit unbinds the capability and marks the pluginFAILED, so a dead plugin stops receiving executions. - Bounded shutdown. The
shutdownRPC and the process reap are each capped; a hung plugin is terminated and then killed.
flowforge_contracts.protocol is the wire contract: newline-delimited JSON,
one JSON-RPC message per line, capped at MAX_MESSAGE_SIZE_BYTES.
Version compatibility is major-locked and minor-forward
(is_protocol_compatible), checked in both directions:
- the plugin refuses a host whose protocol version does not satisfy the
capability's
min_protocol_version; - the in-process host refuses a capability requiring a protocol newer than the host speaks.
Unparseable versions are incompatible — version checking fails closed.
Optional features are advertised in ProtocolCapabilities.features. A parsed
capability set reports only what the peer actually listed, so an absent feature
always reads as unsupported.
execution-context means ExecuteRequest carries the full execution context
(workflow_id, node_id, node_name, attempt, iteration,
workflow_inputs, correlation_id) so an out-of-process node observes the same NodeContext an
in-process node would. Both runtimes build that context through one shared
translator (plugins/node_context.py), so they cannot drift.
scoped-cancel means CancelRequest may name one logical node operation
through correlation_id instead of the whole execution. The distinction is
load-bearing: a node attempt is usually cancelled by its own per-attempt
timeout, and the other nodes of a parallel batch are running in the same
plugin under the same execution_id. Cancelling by execution takes them down
with it, so one slow node fails its whole batch. A request without a
correlation_id still cancels the execution — that is the older wire shape,
and what a genuine execution cancellation means. A plugin that ignores the
field degrades to exactly the old behaviour.
A runtime built on flowforge_sdk gets this for free. A hand-written plugin
that tracks its own in-flight work should key it by correlation_id and
cancel only the matching requests when one is named.
A capability declares what it provides (nodes) and, separately, what it
needs to reach (permissions): network, filesystem, credentials,
environment, subprocess.
This is a declaration and review surface, not a sandbox. FlowOS cannot stop Python code from opening a socket, and the API says so rather than implying a boundary that does not exist. Exactly one permission is enforced:
credentials is enforced. A capability whose nodes declare
required_credentials without requesting the permission is refused at
registration. The SDK also checks it while authoring, but a third-party plugin
can publish any manifest it likes, so the host verifies rather than trusts —
secrets are never handed to a package that did not declare it needs them.
GET /plugins reports declared permissions so an operator can review a
third-party package before installing it. A future sandboxed runtime can
enforce more without changing this vocabulary.
Process isolation is not resource isolation. ResourceLimits records what an
operator asked for, and GET /plugins reports per limit whether it is
actually applied:
| Enforcement | Limits |
|---|---|
enforced (by the runtime) |
max_concurrent_executions, execution_timeout_seconds, startup_timeout_seconds, shutdown_timeout_seconds, max_request_bytes, max_response_bytes |
delegated (needs cgroups, a container, or an orchestrator) |
memory_bytes, cpu_cores |
unset |
anything not requested |
A platform that advertises a memory cap it cannot apply is worse than one with
none: an operator would size a deployment around a guarantee that does not
exist. max_concurrent_executions is one semaphore per capability, so the
limit bounds the plugin rather than each of its node types independently.
Engine-level admission is separate. ConcurrencyLimits bounds node attempts
globally across the worker and, optionally, per execution, so a thousand-way
fan-out becomes a thousand queued coroutines rather than a thousand
simultaneous plugin requests. Queueing happens before dispatch, so a timeout
spent waiting stays a clean, retryable failure.
See also §4 of docs/security.md for storage.
stored credential → CredentialProvider → requirement matching → config["credential"]
→ scope filtering → NodeContext.credentials → ExecuteRequest.credentials → plugin
- A node declares
required_credentialsin its manifest; the capability contract carries them through to the registry. config["credential"]selects which stored credential satisfies a requirement: a string for a single requirement, or a mapping keyed by requirement name. Absent, the requirement's own name is used.- Only the fields listed in
scopesare delivered; empty means the whole payload. - Required credentials fail closed. A missing one fails the node as a configuration error before the plugin is called — the plugin is never given the chance to proceed without the secret.
- Optional credentials are simply omitted.
- A credential of the wrong type does not satisfy a requirement.
- Secrets live only in
NodeContext.credentialsandExecuteRequest.credentials. They are never written to checkpoints, audit records, execution outputs, metrics, or trace attributes.
Some integration nodes accept a plaintext fallback (NotiFly's api_key config
key) for contexts with no credential provider. The resolved credential is
always authoritative and takes precedence; the fallback exists for local
development and is flagged in the editor.
Node configuration is written by workflow authors. An unchecked base_url
would turn any integration node into a general-purpose fetcher able to read
internal HTTP endpoints — bypassing the SSRF guard core.http.request applies.
flowforge_sdk.resolve_endpoint is the shared policy. "The operator decides
what is internal":
- an omitted endpoint uses the operator-configured default (
MINIGOOGLE_URL,NOTIFLY_URL, …), so the localhost development default keeps working; - a configured endpoint on the default host, or on a host listed in
<SERVICE>_ALLOWED_HOSTS, is accepted whatever its address range; - anything else must be a public address. Loopback, link-local, private, multicast, reserved, and unspecified addresses are refused.
Malformed and non-HTTP endpoints are rejected. Every integration node should route its endpoint through this helper.
supported_runtime is honoured, not advisory. The composition root wires a
RuntimeRoutingPluginHost that sends each plugin to the host its manifest asks
for. A capability requesting a runtime the deployment does not provide fails to
start rather than being silently downgraded — quietly running a plugin that
asked for process isolation inside the host process is exactly the false
assurance this design avoids elsewhere.
examples/reference_plugin ships as supported_runtime: process, so the
production path — spawn, handshake, reconcile, dispatch over JSON-RPC,
supervise, shut down — is exercised by the product itself and not only by
tests.
in_process |
process |
|
|---|---|---|
| Declared by | supported_runtime in the manifest |
same |
| Boundary | correctness only | correctness + separate OS process |
| Cost | none | one process per plugin |
| Crash impact | takes the host down | detected, unbound, host survives |
| Use for | first-party and trusted plugins | plugins you do not control |
The same package runs either way with no code change — only the manifest's
supported_runtime and the host wired in create_app differ.
GET /plugins and GET /plugins/{capability_id} answer: what is installed,
from which distribution, at which version, which node types it provides, which
credentials those need, whether it is ready and bound, and when and why
it failed.
status is the lifecycle phase; bound is whether its node types are actually
dispatchable; healthy requires both — a plugin can be READY yet unbound if
its capability lost a node-type conflict.
Prometheus metrics:
flowos_plugin_executions_total{capability,node_type,outcome}—succeeded,failed(the node returned a failure),errored(the dispatch itself raised).flowos_plugin_execution_duration_seconds{capability,node_type}.
Per-node spans carry flowos.workflow.id, flowos.execution.id,
flowos.node.id, and flowos.attempt; see docs/observability.md for the
identifier convention. No configuration value, input, or credential is ever
attached to telemetry.
The reference plugin's tests are the template:
- exercise each node directly against a hand-built
NodeContext; - assert the exact outbound request contract with
httpx.MockTransport; - assert the published capability matches the installed entry points, so a packaging mistake fails a test rather than a deployment.
The platform's own plugin tests worth knowing about:
backend/tests/infrastructure/test_plugin_isolation.py— the isolation loader, including that every workspace plugin actually loads.backend/tests/infrastructure/test_process_plugin.py— real subprocess round trips: a genuine host → subprocess → JSON-RPC → node → host execution, that a process refuses an undeclared node, that it cannot see another installed distribution's nodes, and that a manifest/runtime divergence fails startup.backend/tests/presentation/test_executor_authority.py— one registry, one authority, end to end through a real subprocess.backend/tests/presentation/test_real_process_plugin.py— the real installed reference plugin driven through discovery, spawn, handshake, reconciliation, workflow execution, a genuinekill(), and supervised recovery. No fabricated distributions.backend/tests/application/test_plugin_supervisor.py— restart, backoff, crash loops, and every lifecycle race.backend/tests/application/test_indeterminate_outcomes.py— lost outcomes and effect-gated retries.backend/tests/application/test_backpressure.py— observed peak concurrency under real fan-out.
Live integration suites (examples/*/tests/test_*_integration.py) skip
themselves when the external service is unreachable.
flowforge plugin init ./my-plugin --namespace acme # scaffold
cd my-plugin && pytest # test, no install needed
pip install -e . # register entry points
flowforge plugin validate acme # what the host will check
flowforge plugin inspect acme # what an operator seesvalidate runs the same manifest validation and isolation loading the host
runs at registration, so a green result means the host will genuinely accept
the package. It also works on a manifest JSON file before the package is
installed. inspect shows node effects, credential scopes, permissions, and
runtime — the review surface for a third-party plugin.
The scaffold produces a package that already does everything right: the capability derived from the node manifests, both entry-point groups registered, node effects declared, and a test asserting packaging matches the code. CI scaffolds a fresh plugin and runs its tests on every push, so the authoring loop cannot rot.
Rules that keep an upgrade from being a gamble:
- Protocol — major-locked, minor-forward (
is_protocol_compatible). A capability states the oldest protocol it accepts viamin_protocol_version; both sides check, and an unparseable version is incompatible. - Capability version — reported and reconciled. The handshake must report the same version discovery found; a mismatch is rejected, so a half-upgraded installation cannot serve a manifest the host did not validate.
- Adding a node type is safe: existing workflows are unaffected.
- Removing a node type breaks workflows that reference it — they fail with a clear "no plugin provides node type" configuration error rather than running something unexpected.
- Adding a config field is safe; the editor renders it from the schema. Removing or renaming one is a breaking change for saved workflows.
- Adding a credential requirement requires the
credentialspermission, so the upgrade fails registration rather than silently starting to demand secrets. - Tightening an effect (
idempotent→unsafe) is always safe. Loosening it is a promise about the external service; make it deliberately. - Unknown fields round-trip through
extra, and unknown enum values degrade to the conservative option, so a newer plugin on an older host degrades rather than failing.
Version pinning and migration are deployment concerns FlowOS deliberately does not own yet; nothing in the runtime prevents adding them.