Skip to content

Commit 678f456

Browse files
committed
docs(pipeline): add "Compose Pipelines With Additional Modules" README section and docstring notes
- Add README subsection demonstrating FlowerPowerProject.run and PipelineManager.run with additional_modules (examples, module resolution rules, precedence, reload behavior, and CLI note). - Update PipelineVisualizer docstrings to document additional_modules resolution, reload semantics, and example usages for save_dag/show_dag. - Add doc tasks/proposal files under openspec/changes documenting the docs update and acceptance criteria. - Small docstring tweak in PipelineManager.run to mention additional_modules resolution and precedence.
1 parent a1c243c commit 678f456

File tree

5 files changed

+142
-1
lines changed

5 files changed

+142
-1
lines changed

README.md

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,59 @@ For quick testing or local runs, you can execute your pipeline synchronously. Th
296296
```
297297

298298

299+
### Compose Pipelines With Additional Modules
300+
301+
Hamilton can build a DAG from more than one Python module. FlowerPower surfaces this through the `additional_modules` argument so you can split shared setup, teardown, or utility nodes into their own files.
302+
303+
```python
304+
from flowerpower import FlowerPowerProject
305+
306+
project = FlowerPowerProject.load('.')
307+
308+
# Combine the hello_world pipeline with its setup companion module
309+
result = project.run(
310+
'hello_world',
311+
additional_modules=['pipeline_setup'],
312+
final_vars=["full_greeting"],
313+
)
314+
print(result)
315+
316+
# PipelineManager exposes the same capability
317+
from flowerpower.pipeline import PipelineManager
318+
319+
pm = PipelineManager(base_dir='.')
320+
composed = pm.run(
321+
name='hello_world',
322+
additional_modules=['pipeline_setup'],
323+
final_vars=['full_greeting'],
324+
)
325+
print(composed)
326+
327+
# Visualize the composed DAG
328+
pm.visualizer.save_dag(
329+
name='hello_world',
330+
base_dir='.',
331+
additional_modules=['pipeline_setup'],
332+
)
333+
```
334+
335+
How module resolution works:
336+
337+
- Each string is imported as-is, with a fallback to the underscored variant (replacing `-` with `_`) and to `pipelines.<name>`.
338+
- You can mix strings and already-imported module objects in the same list.
339+
- If multiple modules define the same node, the last module passed to `additional_modules` (and finally the primary pipeline module) wins—matching Hamilton’s precedence rules.
340+
341+
Reload behaviour:
342+
343+
- Setting `reload=True` ensures that every module in `additional_modules` and the main pipeline module is reloaded before execution, making it easy to iterate during development.
344+
345+
Examples:
346+
347+
- The hello-world example demonstrates this pattern: `examples/hello-world/pipelines/setup.py` pairs with `hello_world.py` to provide initialization nodes. The snippets above mirror that structure.
348+
349+
> ℹ️ The Python API currently exposes `additional_modules`. The CLI does not yet forward this option.
350+
351+
299352
## ⚙️ Configuration Overview
300353

301354
FlowerPower uses a layered configuration system:
@@ -381,6 +434,8 @@ print(f"Pipeline config: {info}")
381434
pm.delete('old_pipeline')
382435
```
383436

437+
See [Compose Pipelines With Additional Modules](#compose-pipelines-with-additional-modules) for an example that loads `pipeline_setup` alongside the pipeline module.
438+
384439
**When to use Pipeline-only approach:**
385440
- Simple synchronous workflows
386441
- Testing and development
@@ -409,4 +464,4 @@ You can find the full documentation for FlowerPower, including installation inst
409464

410465
## 📜 License
411466

412-
This project is licensed under the MIT License - see the `LICENSE` file for details. (Placeholder - update with actual license)
467+
This project is licensed under the MIT License - see the `LICENSE` file for details. (Placeholder - update with actual license)
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
## Why
2+
The library now supports composing multiple pipeline modules at runtime via `additional_modules` (e.g., `pm.run("pipeline_1", additional_modules=["setup"])`). We should document this capability so users can discover how to:
3+
- Combine a setup module with a main pipeline
4+
- Understand import/name resolution rules and precedence
5+
- Visualize composed DAGs with the visualizer
6+
7+
## What Changes
8+
- README
9+
- Add a dedicated subsection ("Compose Pipelines With Additional Modules") under Python API usage.
10+
- Provide examples for both `FlowerPowerProject.run` and `PipelineManager.run` using `additional_modules`.
11+
- Mention conflict resolution/precedence (last module wins) and `reload=True` behavior.
12+
- Cross-link to example in `examples/hello-world/`.
13+
- API docstrings
14+
- Ensure `additional_modules` is present and clearly described in `PipelineManager.run(...)` and visualizer methods `save_dag`/`show_dag`.
15+
- Visualizer docs
16+
- Show how to pass `additional_modules` to render a composite DAG.
17+
- Examples
18+
- Reference `examples/hello-world/pipelines/setup.py` alongside `hello_world.py` in docs to illustrate a simple composition path.
19+
- CHANGELOG
20+
- Already updated; verify entry under Unreleased.
21+
22+
Non‑functional/documentation-only change. No API changes.
23+
24+
## Impact
25+
- Files to update:
26+
- README.md (new subsection, examples)
27+
- Optionally add a focused guide: `docs/guide/additional-modules.md` (if guide section exists)
28+
- Verify docstrings for `PipelineManager.run` and `PipelineVisualizer` mention `additional_modules`
29+
- Ensure example references align with `examples/hello-world/pipelines/setup.py`
30+
31+
## Risks / Considerations
32+
- Avoid implying CLI support if CLI doesn’t accept `additional_modules` yet (call out Python API only).
33+
- Clarify module import resolution attempts (`"name"`, `"name" with '-'→'_'`, and `"pipelines.<name>"`).
34+
- Note that duplicate node names resolve by standard Hamilton semantics (later modules override earlier ones).
35+
36+
## Acceptance Criteria
37+
- README contains a clear "Compose Pipelines With Additional Modules" section with runnable snippets.
38+
- `PipelineManager.run` and visualizer method docstrings mention and describe `additional_modules`.
39+
- Example code snippet(s) reference hello-world’s `setup.py` and `hello_world.py` to demonstrate composition.
40+
- No claims about CLI support unless implemented.
41+
- CHANGELOG reflects feature under Unreleased.
42+
43+
## Archival
44+
This is a docs-only change. After deployment, archive with:
45+
`openspec archive docs-add-additional-modules-usage --skip-specs --yes`
46+
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
## 1. Documentation Updates
2+
- [x] 1.1 README: add "Compose Pipelines With Additional Modules" section with examples for `project.run(...)` and `pm.run(...)` using `additional_modules=["setup"]`.
3+
- [x] 1.2 README: document precedence (last module passed wins on duplicate node names) and `reload=True` behavior (reloads all loaded modules).
4+
- [x] 1.3 Visualizer docs: show `additional_modules` usage in `save_dag`/`show_dag` examples.
5+
- [x] 1.4 Cross-reference examples: highlight `examples/hello-world/pipelines/setup.py` with `hello_world.py` for composition.
6+
- [x] 1.5 API docstrings: verify/expand `additional_modules` param documentation in `PipelineManager.run` and `PipelineVisualizer` methods.
7+
- [x] 1.6 Confirm CHANGELOG Unreleased entry includes this feature.
8+
9+
## 2. Optional Guides (if docs site in use)
10+
- [ ] 2.1 Create `docs/guide/additional-modules.md` with deeper explanation (import resolution, tips, troubleshooting, best practices).
11+
12+
## 3. Validation
13+
- [x] 3.1 Render README locally to verify code blocks and formatting.
14+
- [ ] 3.2 (If docs site) run docs build locally to ensure no broken links.
15+
16+
## 4. Rollout
17+
- [ ] 4.1 Submit PR for review.
18+
- [ ] 4.2 After merge/deploy, archive change with `openspec archive docs-add-additional-modules-usage --skip-specs --yes`.

src/flowerpower/pipeline/manager.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,11 @@ def run(
381381
additional_modules (list[str | ModuleType] | None): Extra modules to
382382
load alongside the primary pipeline for Hamilton execution.
383383
Example: ["setup", setup_module]
384+
Notes:
385+
- String entries are attempted as-is, with fallbacks to a
386+
hyphen-to-underscore variant and to `pipelines.<name>`.
387+
- When multiple modules provide the same node, later modules
388+
take precedence following Hamilton semantics.
384389
reload (bool): Force reload of pipeline configuration.
385390
log_level (str | None): Logging level for the execution. Default None uses project config.
386391
Valid values: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"

src/flowerpower/pipeline/visualizer.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,13 @@ def _get_dag_object(
4747
Raises:
4848
ImportError: If the module cannot be loaded.
4949
50+
Notes:
51+
- String entries in ``additional_modules`` are resolved using the same
52+
strategy as pipeline execution: the raw name, a hyphen-to-underscore
53+
variant, and ``pipelines.<name>`` are attempted in order.
54+
- ``reload=True`` reloads every module (primary + additional) before the
55+
driver is constructed, which mirrors runtime execution behaviour.
56+
5057
"""
5158
# Load pipeline-specific config
5259
pipeline_cfg = PipelineConfig.load(name=name, fs=self._fs)
@@ -90,6 +97,10 @@ def save_dag(
9097
Raises:
9198
ImportError: If any module cannot be loaded.
9299
100+
Notes:
101+
- ``additional_modules`` uses the same resolution strategy as pipeline execution.
102+
- Pass ``reload=True`` to reload the primary and additional modules before rendering.
103+
93104
Example:
94105
>>> from flowerpower.pipeline.visualizer import PipelineVisualizer
95106
>>> visualizer = PipelineVisualizer(project_cfg, fs)
@@ -99,6 +110,7 @@ def save_dag(
99110
... format="png",
100111
... additional_modules=["setup"],
101112
... )
113+
>>> # Compose hello_world + setup from examples/hello-world/pipelines/
102114
"""
103115
dag = self._get_dag_object(
104116
name=name, reload=reload, additional_modules=additional_modules
@@ -160,6 +172,10 @@ def show_dag(
160172
Raises:
161173
ImportError: If any module cannot be loaded.
162174
175+
Notes:
176+
- ``additional_modules`` follows the same resolution strategy as pipeline execution.
177+
- Use ``reload=True`` to pull in code changes from every module before rendering.
178+
163179
Example:
164180
>>> from flowerpower.pipeline.visualizer import PipelineVisualizer
165181
>>> visualizer = PipelineVisualizer(project_cfg, fs)
@@ -169,6 +185,7 @@ def show_dag(
169185
... format="png",
170186
... additional_modules=["setup"],
171187
... )
188+
>>> # Compose hello_world + setup modules during development
172189
"""
173190
dag = self._get_dag_object(
174191
name=name, reload=reload, additional_modules=additional_modules

0 commit comments

Comments
 (0)