Skip to content

Commit 3bf32fc

Browse files
erichareerichare
committed
Add seed-directory extension loading and docs
Introduce filesystem "seed directory" extension support and authoring docs. Adds three documentation pages (quickstart, manifest reference, author guide) and wires them into the docs sidebar. Implement load_seed_extensions to discover/load bundles from $LANGFLOW_SEED_DIR (default /opt/langflow/bundles), export it from loader/__init__ and extension package, and integrate it into the components import pipeline. Handle installed-vs-seed shadowing by preferring installed distributions and appending a typed seed-bundle-shadowed ExtensionError; add the corresponding error code and message. Update schema doc link and add unit/integration tests to cover seed loading, determinism, shadowing behavior, and migration-target resolution.
1 parent 4eff504 commit 3bf32fc

14 files changed

Lines changed: 950 additions & 3 deletions

File tree

‎docs/docs/Develop/extensions-author-guide.mdx‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
---
2+
title: Author guide for Langflow Extensions
3+
slug: /extensions-author-guide
4+
---
5+
6+
This guide covers how to design, build, and ship a Langflow Extension Bundle from scratch, plus the conventions that make the result safe to maintain alongside the rest of Langflow. If you just want to get something running, start with [Build your first Langflow Extension](/extensions-quickstart) — that's the three-minute path. Read this when you're ready to publish, port a component out of `lfx.components.<provider>`, or evaluate whether your extension idea fits the v0 contract.
7+
8+
:::info First-delivery scope
9+
This page documents the first-delivery slice of the Bundle Separation epic (LE-905). It covers the foundation (manifest, single-Bundle loader, atomic-swap reload) plus the pilot migration pattern. The wider Extension System vision — services, routes, registries, partner handover — is described in the project's internal design docs and ships in later epics.
10+
:::
11+
12+
## Concepts
13+
14+
Three terms appear constantly in the codebase and the manifest schema. Get them straight before reading further:
15+
16+
* **Component** — a `Component` subclass with `build()`, `inputs`, and `outputs`. Same definition as in pre-extensions Langflow.
17+
* **Bundle** — a named group of components, addressed at runtime as `ext:<bundle>:<Class>@<slot>`. Bundles are the unit of palette grouping and reload.
18+
* **Extension** — the **distribution** that ships a Bundle plus a manifest. This is what you `pip install`. v0 requires exactly one Bundle per Extension.
19+
20+
The runtime "slot" tells you where a Bundle came from. There are two:
21+
22+
* `@official` — installed pip distributions, the seed directory (`/opt/langflow/bundles` or `$LANGFLOW_SEED_DIR`), and Extensions registered through `lfx extension dev`.
23+
* `@extra` — loose subdirectories under `LANGFLOW_COMPONENTS_PATH`. Useful for ad-hoc local dev that you don't want to package.
24+
25+
The full vocabulary lives in [`BUNDLE_API.md`](https://github.com/langflow-ai/langflow/blob/main/BUNDLE_API.md). That document is the stable surface — anything outside it is internal, and your bundle should not import it.
26+
27+
## Decide whether to ship as an Extension
28+
29+
You should ship as an Extension when:
30+
31+
* The component's runtime dependencies are heavy enough that bundling them with everyone's `pip install langflow` is wasteful.
32+
* The release cadence differs from Langflow core (you want to fix a bug in your component without waiting for a Langflow release).
33+
* You want to own the issue tracker and the PR review for the component.
34+
35+
Stick with an in-tree component when:
36+
37+
* The component is part of a tightly-integrated set with the rest of `lfx.components.*` and would have circular dependencies if extracted.
38+
* The runtime dependencies are already in Langflow's transitive set.
39+
40+
If you're unsure, the deciding criterion in practice is "would I want to release this independently of Langflow?". If the answer is yes, extract it.
41+
42+
## Repository layout
43+
44+
The convention every shipped bundle follows (the DuckDuckGo pilot is the reference: see [`src/bundles/duckduckgo`](https://github.com/langflow-ai/langflow/tree/main/src/bundles/duckduckgo)):
45+
46+
```
47+
my-extension/
48+
├── README.md
49+
├── extension.json
50+
├── pyproject.toml
51+
└── src/
52+
└── lfx_my_extension/
53+
├── __init__.py
54+
├── extension.json
55+
└── components/
56+
└── my_bundle/
57+
├── __init__.py
58+
└── my_component.py
59+
```
60+
61+
Why two `extension.json` files? The outer one is for the developer running `lfx extension validate` against the source tree. The inner one is the copy that ships **inside the wheel**, so `importlib.metadata.files()` can find it after `pip install`. The two should stay byte-identical; the porting script keeps them in sync.
62+
63+
The component path declared in `extension.json` (`bundles[].path`) is the inner `components/<bundle_name>/` directory. Keeping that name aligned with the bundle name lets saved flows that reference the legacy import path migrate cleanly via a single migration-table entry. See [Manifest reference](/extensions-manifest) for the full set of fields.
64+
65+
## Authoring rules
66+
67+
A few constraints turn into hard CI gates:
68+
69+
1. **Import only from `lfx.*`.** The bundle is installed against the public `BUNDLE_API` surface, not Langflow internals. `from langflow...` is rejected by validate. Run `grep -r "from langflow" src/lfx_my_extension/` before pushing.
70+
2. **Bundle module imports must be side-effect-free.** Top-level I/O (file reads, network calls, `print`) is rejected with `top-level-io-disallowed`. If you need expensive setup, do it in `Component.build()`.
71+
3. **No wildcard imports at module top-level** (`from foo import *`). Validate flags this with `import-star-disallowed`.
72+
4. **Every Component class must declare `build()`.** Stub it with `return None` if you're scaffolding; the validator surfaces `build-method-missing` otherwise.
73+
5. **Bundle names are unique.** If your bundle name collides with an existing one (e.g. shipping a second `openai` bundle), the loader keeps the first-discovered one and emits `duplicate-component-name` for any colliding class names.
74+
75+
Each rule is enforced by a typed error code with a concrete `hint`. Run `lfx extension validate .` early and often — the validator is intentionally cheap so it fits in a pre-commit hook.
76+
77+
## Migration table for moved components
78+
79+
If your Extension extracts a component that was previously in-tree (i.e. anyone might have a saved flow referencing the old path), add a migration-table entry in the same PR that lands the extraction. The table lives at [`src/lfx/src/lfx/extension/migration/migration_table.json`](https://github.com/langflow-ai/langflow/blob/main/src/lfx/src/lfx/extension/migration/migration_table.json) and is **append-only** — CI rejects any removal.
80+
81+
Three legacy reference forms each get their own entry:
82+
83+
```json
84+
{
85+
"import_path": "lfx.components.duckduckgo.duck_duck_go_search_run.DuckDuckGoSearchComponent",
86+
"target": "ext:duckduckgo:DuckDuckGoSearchComponent@official",
87+
"added_in": "1.10.0"
88+
}
89+
```
90+
91+
| Legacy form | Always added? | Why |
92+
| --- | --- | --- |
93+
| Full import path (`lfx.components.foo.bar.Class`) | Yes | The historical canonical reference in saved flows. |
94+
| Package import path (`lfx.components.foo.Class`) | Yes | Some flows referenced the package re-export. |
95+
| Bare class name (`Class`) | Only if globally unique | A bare name can match more than one bundle; ambiguous names go in `ambiguous_bare_names` instead and surface `component-name-ambiguous` so the user gets a fix hint instead of the wrong component. |
96+
| Pre-Phase-A slot (`ext:foo:Class@official-pre-a`) | Yes | Reserved for future phases that change the slot layout. |
97+
98+
CI enforces bare-name uniqueness via [`scripts/migrate/check_bare_names.py`](https://github.com/langflow-ai/langflow/blob/main/scripts/migrate/check_bare_names.py). Add the entry once, run the check locally, and the deserializer will rewrite saved flows on load.
99+
100+
## Reload semantics
101+
102+
Atomic-swap reload (Mode A only) is what makes the dev loop tolerable. The five stages:
103+
104+
1. **Stage 1 — load** the new bundle into a `__reload_staging__.<id>` namespace so `sys.modules` for the live bundle is untouched.
105+
2. **Stage 2 — validate** the staging load. Any error aborts here; the live bundle is unchanged.
106+
3. **Stage 3 — swap** under the registry write-lock. New flows resolve to the new class; in-flight flows keep their pre-swap class reference.
107+
4. **Stage 4 — clean up** the staging namespace.
108+
5. **Stage 5 — emit** a `bundle_reloaded` event with the added/removed component lists.
109+
110+
Reload is **not** a trust boundary. Bundle code was trusted at install time; the reload pipeline only handles the swap mechanics. Mode B/C deployments rebuild the Docker image rather than reload — see [Production install pattern for Extensions](/deployment-extensions-production).
111+
112+
## Porting an in-tree component
113+
114+
The mechanical steps for extracting `src/lfx/src/lfx/components/<provider>/` into a standalone bundle live in [`src/bundles/PORTING.md`](https://github.com/langflow-ai/langflow/blob/main/src/bundles/PORTING.md). That's the canonical recipe; this page covers the design choices around it. The DuckDuckGo extraction (LE-1023) is the reference implementation — diff `src/bundles/duckduckgo/` against any in-tree component to see the shape.
115+
116+
A few things that surprise people:
117+
118+
* **The user-visible `pip install langflow` does not change.** The metapackage adds the new bundle as a regular pip dependency, so a flat `pip install langflow` keeps the same component set as before. Decoupling that (where you'd opt into bundles) is a later milestone.
119+
* **You can ship before the in-tree copy is removed.** The extracted bundle gets `@official` precedence; the in-tree copy stays as a fallback during the deprecation window so users on older saved flows are never broken.
120+
* **Bare-name uniqueness is the most common gotcha.** Components like `MergeDataComponent`, `SplitTextComponent`, and `SubFlowComponent` already exist in multiple bundles. Run the bare-names check before opening the PR.
121+
122+
## Publishing checklist
123+
124+
Before merging the extraction PR or releasing a fresh bundle:
125+
126+
1. `lfx extension validate <path>` exits zero against your manifest.
127+
2. `pytest src/bundles/<bundle>/tests` passes.
128+
3. Migration table entries cover every legacy form (full path, package path, bare name if unique, pre-A slot).
129+
4. `scripts/migrate/check_bare_names.py` passes.
130+
5. `scripts/migrate/check_migration_append_only.py --base origin/main` passes.
131+
6. `scripts/migrate/check_bundle_api_changelog.py --base origin/main` passes if you touched the `BUNDLE_API.md` surface.
132+
7. The dogfood checklist (template at [`src/bundles/duckduckgo/M1_DOGFOOD_CHECKLIST.md`](https://github.com/langflow-ai/langflow/blob/main/src/bundles/duckduckgo/M1_DOGFOOD_CHECKLIST.md)) is filled in by an engineer who is **not** on the Extension team. Save a flow on the pre-migration release, upgrade, confirm it loads and runs identically.
133+
8. The PR description links to the migration-table entries and the dogfood evidence.
134+
135+
If any step fails, the extraction is not done. The dogfood gate especially is non-negotiable — the foundation tests prove the migration table rewrites references; only an end-to-end run proves the rewritten flow still does what the user expects.
136+
137+
## See also
138+
139+
* [Build your first Langflow Extension](/extensions-quickstart) — three-minute scaffold-and-run path.
140+
* [Manifest reference](/extensions-manifest) — every field accepted by `extension.json`.
141+
* [Production install pattern for Extensions](/deployment-extensions-production) — Mode B/C Docker image patterns and the seed-directory alternative.
142+
* [`BUNDLE_API.md`](https://github.com/langflow-ai/langflow/blob/main/BUNDLE_API.md) — the stable surface bundle code may consume.
143+
* [`src/bundles/PORTING.md`](https://github.com/langflow-ai/langflow/blob/main/src/bundles/PORTING.md) — the step-by-step porting recipe.

‎docs/docs/Develop/extensions-manifest.mdx‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
---
2+
title: Manifest reference
3+
slug: /extensions-manifest
4+
---
5+
6+
This page is the field-by-field reference for the `extension.json` manifest the Langflow Extension loader consumes. The canonical JSON Schema lives at [`https://schemas.langflow.org/extension/v1.json`](https://schemas.langflow.org/extension/v1.json) and is generated from the Pydantic model at [`lfx.extension.manifest.ExtensionManifest`](https://github.com/langflow-ai/langflow/blob/main/src/lfx/src/lfx/extension/manifest.py). To export the live schema:
7+
8+
```bash
9+
lfx extension schema --output extension.schema.json
10+
```
11+
12+
Use the `$schema` reference in your manifest so editors can autocomplete and validate inline:
13+
14+
```json
15+
{
16+
"$schema": "https://schemas.langflow.org/extension/v1.json",
17+
"id": "lfx-my-extension",
18+
"version": "0.1.0",
19+
"name": "My extension",
20+
"lfx": { "compat": ["1"] },
21+
"bundles": [{ "name": "my_bundle", "path": "components/my_bundle" }]
22+
}
23+
```
24+
25+
## Top-level fields
26+
27+
| Field | Type | Required | Description |
28+
| --- | --- | --- | --- |
29+
| `id` | string | yes | Globally-unique extension ID. Lowercase, hyphenated, starts with a letter, 2-64 chars. Convention: `lfx-<provider>`. |
30+
| `version` | string | yes | SemVer 2.0.0 version string for this extension release. |
31+
| `name` | string | yes | Human-readable display name shown in the Langflow palette. 1-200 chars. |
32+
| `description` | string \| null | no | Optional one-paragraph summary, max 2000 chars. |
33+
| `lfx` | object | yes | [Compatibility declaration](#lfx-compatibility-declaration) against the BUNDLE_API contract. |
34+
| `bundles` | array | yes | [Bundle list](#bundles). v0 accepts **exactly one** bundle. |
35+
| `capabilities` | object | no | [Optional capability flags](#capabilities). Defaults to all-false. |
36+
| `$schema` | string | no | Optional pointer to this JSON Schema; editors use it for autocomplete. |
37+
38+
`additionalProperties: false` — any field not listed here is rejected with a typed error. Reserved names (`services`, `routes`, `hooks`, `starterProjects`, `userConfig`) are documented under [Deferred fields](#deferred-fields) and surface a more specific error code.
39+
40+
## `lfx`: compatibility declaration
41+
42+
```json
43+
"lfx": { "compat": ["1"] }
44+
```
45+
46+
| Field | Type | Description |
47+
| --- | --- | --- |
48+
| `compat` | array of strings | Non-empty list of `BUNDLE_API.md` contract versions this extension supports. Each entry is a positive-integer string. |
49+
50+
The runtime compares `str(BUNDLE_API_VERSION)` against this list. A mismatch fails install with `version-constraint-unsatisfied`. v0 accepts only `"1"`; lists like `["1", "2"]` become meaningful when a future BUNDLE_API revision ships.
51+
52+
## `bundles`
53+
54+
```json
55+
"bundles": [
56+
{ "name": "my_bundle", "path": "components/my_bundle" }
57+
]
58+
```
59+
60+
| Field | Type | Description |
61+
| --- | --- | --- |
62+
| `name` | string | Bundle name; addressable as `ext:<name>:<Class>@<slot>`. Lowercase snake_case, starts with a letter, 2-64 chars. |
63+
| `path` | string | Path to the bundle directory, relative to the manifest. Must not start with `/` or contain `..`. |
64+
65+
v0 enforces `minItems: 1, maxItems: 1`; multi-bundle extensions are rejected with `multi-bundle-deferred-in-this-milestone` and ship in a later epic.
66+
67+
## `capabilities`
68+
69+
Optional. Defaults to `{ "requiresCredentials": false }`.
70+
71+
| Field | Type | Description |
72+
| --- | --- | --- |
73+
| `requiresCredentials` | bool | If true, the loader records that components in this bundle expect credential variables to be configured before use. |
74+
75+
Additional capability keys are rejected with `extra="forbid"` so a misspelled key surfaces immediately rather than silently turning a feature off.
76+
77+
## Deferred fields
78+
79+
The schema strips these names from the published `properties` map but reserves them via [`x-deferred-fields`](https://schemas.langflow.org/extension/v1.json), so a manifest that sets one gets a specific error code instead of the generic "additional property" message.
80+
81+
| Reserved key | Replacement error code | Future epic |
82+
| --- | --- | --- |
83+
| `services` | `field-deferred-in-this-milestone` | B2 — non-component primitives |
84+
| `routes` | `field-deferred-in-this-milestone` | B2 — non-component primitives |
85+
| `hooks` | `field-deferred-in-this-milestone` | B2 — non-component primitives |
86+
| `starterProjects` | `field-deferred-in-this-milestone` | later milestone |
87+
| `userConfig` | `field-deferred-in-this-milestone` | later milestone |
88+
89+
A manifest that sets any of these to a non-null value is rejected at validate / load time. Setting them to `null` is allowed (the loader treats null and absent identically).
90+
91+
## Pyproject alternative
92+
93+
If you'd rather not ship a separate `extension.json` next to `pyproject.toml`, declare the same fields under `[tool.langflow.extension]`:
94+
95+
```toml
96+
[tool.langflow.extension]
97+
id = "lfx-my-extension"
98+
version = "0.1.0"
99+
name = "My extension"
100+
101+
[tool.langflow.extension.lfx]
102+
compat = ["1"]
103+
104+
[[tool.langflow.extension.bundles]]
105+
name = "my_bundle"
106+
path = "components/my_bundle"
107+
```
108+
109+
The loader prefers an `extension.json` when both exist. The Pydantic validator is the same so error codes and field semantics are identical.
110+
111+
## Error codes raised against this manifest
112+
113+
The loader and validator both emit typed errors keyed by the manifest field that triggered them. The full code list is at [`lfx.extension.errors.ERROR_CODES`](https://github.com/langflow-ai/langflow/blob/main/src/lfx/src/lfx/extension/errors.py); the codes most relevant when authoring a manifest are:
114+
115+
| Code | Cause |
116+
| --- | --- |
117+
| `manifest-invalid` | Schema validation failed; the message names the field. |
118+
| `manifest-not-found` | No `extension.json` and no `[tool.langflow.extension]` section at the extension root. |
119+
| `version-constraint-unsatisfied` | `lfx.compat` does not include this Langflow's `BUNDLE_API_VERSION`. |
120+
| `field-deferred-in-this-milestone` | A reserved field was set to a non-null value. |
121+
| `multi-bundle-deferred-in-this-milestone` | `bundles` has more than one entry. |
122+
| `path-escape` | A `bundles[].path` resolves outside the manifest root (typically a symlink). |
123+
| `bundle-path-not-found` | `bundles[].path` does not exist or is not a directory. |
124+
125+
Run `lfx extension validate <path>` to see every error as a structured object with `code`, `message`, `location`, `hint`, and `ref_url`.
126+
127+
## See also
128+
129+
* [Build your first Langflow Extension](/extensions-quickstart) — three-minute getting-started.
130+
* [Author guide for Langflow Extensions](/extensions-author-guide) — long-form authoring guidance, including how to port an existing in-tree component.
131+
* [Production install pattern for Extensions](/deployment-extensions-production) — Mode B/C Docker image patterns.
132+
* [`BUNDLE_API.md`](https://github.com/langflow-ai/langflow/blob/main/BUNDLE_API.md) — the stable surface bundle code may consume.

0 commit comments

Comments
 (0)