Commit e5876f1
authored
fix(python): load dynamic plugin specs from TOML (#694)
#### Overview
Adds a minimal Python compatibility API that converts standard `[[plugins.dynamic]]` records from one explicit `plugins.toml` into the existing `DynamicPluginActivationSpec` objects accepted by `initialize_with_dynamic_plugins()`.
This unblocks Python applications that embed Relay without introducing the larger file-backed activation, lifecycle reconciliation, dynamic layering, or initialization redesign proposed for a later release. The new API is intentionally a temporary 0.7 surface:
```python
plugin_config_path = os.environ["NEMO_RELAY_PLUGINS_TOML"]
dynamic_plugins = plugin.load_dynamic_plugin_activation_specs(plugin_config_path)
activation = await plugin.initialize_with_dynamic_plugins({}, dynamic_plugins)
```
`NEMO_RELAY_PLUGINS_TOML` is an optional host-side convention in this example. Relay does not read the environment variable automatically; the embedding application resolves a path through its environment, command-line, or configuration system and passes that path to the helper.
- [X] I confirm this contribution is my own work, or I have the right to submit it under this project's license.
- [X] I searched existing issues and open pull requests, and this does not duplicate existing work.
A broader implementation exists in #684. This PR is a deliberately scoped 0.7 alternative that reuses the existing activation owner instead of introducing shared lifecycle and host-configuration infrastructure.
#### Details
##### Public API
Adds:
```python
def load_dynamic_plugin_activation_specs(
plugin_config_path: str | os.PathLike[str],
) -> list[DynamicPluginActivationSpec]: ...
```
The helper:
* Reads one explicitly selected `plugins.toml`.
* Parses every `[[plugins.dynamic]]` record in declaration order.
* Resolves relative manifest paths against the selected file.
* Reads `plugin.id` and `plugin.kind` from each manifest.
* Preserves the record's JSON-compatible `config`.
* Rejects malformed TOML, invalid record shapes, unsupported fields, invalid plugin identities, duplicate plugin IDs, and non-JSON configuration.
* Returns the existing activation-spec type without loading code.
The existing dynamic initializer now accepts a `Sequence` rather than only a `list`. This reflects its existing behavior and allows parser results, lists, and tuples to compose without casts.
##### Developer flow
```mermaid
flowchart LR
User["User selects a plugins.toml"] -->
Host["Embedding host resolves the path"]
Env["Optional NEMO_RELAY_PLUGINS_TOML"] --> Host
Host --> Helper["load_dynamic_plugin_activation_specs(path)"]
Helper --> Config["Read one explicit plugins.toml"]
Config --> Records["Parse [[plugins.dynamic]] records"]
Records --> Manifests["Resolve and read relay-plugin.toml manifests"]
Manifests --> Specs["Build DynamicPluginActivationSpec list"]
Specs --> Initialize["initialize_with_dynamic_plugins(config, specs)"]
Initialize --> Activation["Owned PluginHostActivation"]
Activation --> Runtime["Host retains activation while work is admitted"]
Runtime --> Close["await activation.close() during shutdown"]
```
##### Configuration behavior
The temporary dynamic path and existing static configuration path remain separate:
```mermaid
flowchart TB
subgraph Static["Existing static component resolution"]
UserConfig["User plugins.toml"] --> StaticLayering["User → project → system → programmatic overlay"]
ProjectConfig["Project .nemo-relay/plugins.toml"] --> StaticLayering
SystemConfig["System /etc/nemo-relay/plugins.toml"] --> StaticLayering
end
subgraph Dynamic["New 0.7 compatibility path"]
ExplicitPath["One explicit plugins.toml path"] --> DynamicParser["Parse [[plugins.dynamic]] only"]
DynamicParser --> DynamicSpecs["Explicit activation specs"]
end
StaticLayering --> HostInitializer["Existing dynamic host initializer"]
DynamicSpecs --> HostInitializer
HostInitializer --> OwnedHost["PluginHostActivation"]
```
The helper does not perform dynamic-plugin layering. It reads only the explicitly supplied file. Static `[[components]]` from that file are inherited only when the same file is also selected by Relay's normal static discovery.
Every dynamic declaration in the selected file becomes an activation spec. Passing those specs to `initialize_with_dynamic_plugins()` is explicit consent to load the referenced trusted native libraries or worker processes.
Python workers that require a lifecycle-managed `environment_ref` still require the existing explicit activation or CLI lifecycle path.
##### Intentional non-goals
This PR does not:
* Consolidate `initialize()` and `initialize_with_dynamic_plugins()`.
* Add a unified `initialize_from_plugins_toml()` API.
* Discover or merge dynamic records across user, project, and system layers.
* Read or reconcile `.dynamic-plugins.json`.
* Consult CLI enablement or tombstone state.
* Provision or attest Python worker environments.
* Change plugin enablement, install plugins, or execute package managers.
* Change Rust, Node.js, Go, FFI, or CLI behavior.
The helper is documented as a 0.7 compatibility surface and is expected to be deprecated after the unified file-backed initializer lands. Keeping the conversion behind one Relay API lets embedded hosts remove their TOML and manifest parsing now while keeping the future migration localized to one call site.
##### Documentation and validation
Updates the Python type stub, plugin-configuration guide, and 0.7 release notes. Tests cover relative and absolute manifest resolution, native and worker spec construction, config preservation, malformed records and TOML, missing manifests, duplicate IDs, and end-to-end native activation from a real `[[plugins.dynamic]]` record.
Validation completed:
* Focused parser and native-activation tests: `16 passed`.
* Ruff formatting and linting.
* `ty` type checking.
* Changed-file and repository-wide pre-commit suites.
* Cargo formatting, clippy, check, and dependency-policy checks.
* Python worker protobuf compatibility.
* Go formatting and vet.
* Node formatting and public docstring checks.
* Fern structure and strict broken-link validation.
The complete dynamic-host Python module was also attempted locally. Pre-existing tests inherited an invalid machine-level `/etc/nemo-relay/plugins.toml`, and sandboxed worker tests could not bind Unix sockets. The tests directly covering this change passed independently.
Breaking changes: none.
#### Where should the reviewer start?
Start with `python/nemo_relay/plugin.py`, specifically `load_dynamic_plugin_activation_specs()`.
The central design decision is that this helper performs only the missing file-to-activation-spec conversion. It deliberately reuses the existing dynamic initializer and owned activation lifetime rather than introducing another activation owner or pulling CLI lifecycle behavior into the Python binding.
Then review `python/tests/test_dynamic_plugin_host.py` for the standard TOML parsing, failure behavior, and end-to-end native activation coverage.
#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
* Relates to #673
* Relates to #684
* Relates to [NousResearch/hermes-agent#77915](<NousResearch/hermes-agent#77915>)
## Summary by CodeRabbit
* **New Features**
* Added a Python compatibility helper for loading dynamic plugin activation specifications from a selected `plugins.toml` file.
* Supports manifest path resolution, ordered activation specifications, nested configuration, duplicate detection, and validation of plugin records and JSON values.
* Dynamic plugin initialization now accepts any ordered collection of activation specifications.
* **Documentation**
* Added guidance covering configuration resolution, explicit loading consent, supported behavior, limitations, and planned deprecation.
* **Tests**
* Expanded coverage for valid configurations, absolute and nested manifest paths, malformed files, duplicate IDs, and missing manifests.
## Summary by CodeRabbit
* **New Features**
* Added support for loading dynamic plugin activation settings from a selected `plugins.toml` file.
* Added validation for manifests, duplicate identifiers, malformed configuration, and invalid JSON values.
* Dynamic plugin initialization now accepts any ordered collection of activation specifications.
* **Documentation**
* Added configuration guidance, behavior details, limitations, compatibility notes, and planned deprecation information.
* **Tests**
* Added coverage for valid configurations, path resolution, nested settings, and common loading errors.
Authors:
- Bryan Bednarski (https://github.com/bbednarski9)
Approvers:
- Will Killian (https://github.com/willkill07)
- Maryam Najafian (https://github.com/mnajafian-nv)
URL: #6941 parent 05700e1 commit e5876f1
5 files changed
Lines changed: 345 additions & 4 deletions
File tree
- docs
- about-nemo-relay/release-notes
- configure-plugins
- python
- nemo_relay
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
105 | 111 | | |
106 | 112 | | |
107 | 113 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
166 | 210 | | |
167 | 211 | | |
168 | 212 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
14 | 18 | | |
15 | 19 | | |
| 20 | + | |
16 | 21 | | |
17 | 22 | | |
18 | 23 | | |
| |||
417 | 422 | | |
418 | 423 | | |
419 | 424 | | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
420 | 533 | | |
421 | 534 | | |
422 | 535 | | |
| |||
452 | 565 | | |
453 | 566 | | |
454 | 567 | | |
455 | | - | |
| 568 | + | |
456 | 569 | | |
457 | 570 | | |
458 | 571 | | |
| |||
607 | 720 | | |
608 | 721 | | |
609 | 722 | | |
| 723 | + | |
610 | 724 | | |
611 | 725 | | |
612 | 726 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
160 | 161 | | |
161 | 162 | | |
162 | 163 | | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
163 | 167 | | |
164 | 168 | | |
165 | 169 | | |
166 | 170 | | |
167 | | - | |
| 171 | + | |
168 | 172 | | |
169 | 173 | | |
170 | 174 | | |
| |||
0 commit comments