Skip to content

Commit a1355fb

Browse files
legoutlegout
committed
docs(guide): add "Asynchronous Execution" guide; update nav and openspec
- Add new guide: docs/docs/guide/async-execution.md (run_async examples, RunConfig.async_driver, additional_modules, requirements, troubleshooting) - Cross-link new guide from advanced.md and index.md - Update docs/mkdocs.yml to include guide in navigation - Add openspec archive/spec/tasks for async-driver update and docs-add-async-execution-usage (proposals, specs, and task checklists) - Bump pyproject version to 0.34.1
1 parent 476e61d commit a1355fb

File tree

11 files changed

+187
-3
lines changed

11 files changed

+187
-3
lines changed

‎docs/docs/advanced.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22

33
Welcome to the advanced usage guide for FlowerPower. This document covers more complex configurations and use cases to help you get the most out of the library.
44

5+
See also:
6+
7+
- Guides → [Compose Pipelines With Additional Modules](guide/additional-modules.md)
8+
- Guides → [Asynchronous Execution](guide/async-execution.md)
9+
510
## Configuration Flexibility
611

712
FlowerPower offers multiple ways to configure your project, ensuring flexibility for different environments and workflows. Configuration is applied in this order (highest wins):
@@ -168,4 +173,4 @@ Here are some common issues and how to resolve them:
168173
* **Module Not Found**: Make sure your pipeline and task modules are in Python's path. You can add directories to the path using the `PYTHONPATH` environment variable.
169174

170175
!!! note
171-
For more detailed information, refer to the API documentation.
176+
For more detailed information, refer to the API documentation.

‎docs/docs/guide/async-execution.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Asynchronous Execution
2+
3+
FlowerPower integrates with Hamilton’s asynchronous driver to let you await
4+
pipeline runs inside event loops (e.g., FastAPI, notebooks, asyncio scripts).
5+
6+
This page explains how to use `PipelineManager.run_async`, what the
7+
`RunConfig.async_driver` toggle does, and how async execution parallels the
8+
behaviour of synchronous runs.
9+
10+
## Quick start
11+
12+
```python
13+
import asyncio
14+
from flowerpower.pipeline import PipelineManager
15+
16+
pm = PipelineManager(base_dir='.')
17+
18+
async def run_async_pipeline():
19+
result = await pm.run_async(
20+
'hello_world',
21+
final_vars=['full_greeting'],
22+
)
23+
print(result['full_greeting'])
24+
25+
asyncio.run(run_async_pipeline())
26+
```
27+
28+
### With additional modules
29+
30+
You can compose multiple modules in async mode just like sync mode using
31+
`additional_modules`:
32+
33+
```python
34+
async def run_async_composed():
35+
composed = await pm.run_async(
36+
'hello_world',
37+
additional_modules=['pipeline_setup'],
38+
final_vars=['full_greeting'],
39+
)
40+
print(composed['full_greeting'])
41+
```
42+
43+
## RunConfig.async_driver
44+
45+
- `RunConfig.async_driver` controls whether the async driver is used.
46+
- Default behaviour when calling `run_async(...)` is to use the async driver.
47+
- Setting `async_driver=False` raises a `ValueError` – use `pm.run(...)` for
48+
synchronous execution instead.
49+
50+
```python
51+
from flowerpower.cfg.pipeline.run import RunConfig
52+
53+
# Explicitly opt in (usually not necessary for run_async)
54+
cfg = RunConfig(async_driver=True)
55+
await pm.run_async('hello_world', run_config=cfg)
56+
57+
# Opting out raises an error (use pm.run)
58+
cfg = RunConfig(async_driver=False)
59+
try:
60+
await pm.run_async('hello_world', run_config=cfg)
61+
except ValueError:
62+
# Fall back to sync
63+
pm.run('hello_world')
64+
```
65+
66+
## Parity with synchronous runs
67+
68+
Async runs honour the same flags and behaviours:
69+
70+
- `reload=True` reloads the main and additional modules before executing.
71+
- `log_level` configures logging for the run.
72+
- Adapters and adapter configs are applied the same as in sync mode.
73+
74+
## Requirements
75+
76+
Async execution relies on `hamilton.async_driver`. Ensure your Hamilton version
77+
provides it, and upgrade if necessary:
78+
79+
```bash
80+
pip install -U hamilton
81+
```
82+
83+
## Troubleshooting
84+
85+
| Symptom | Cause | Fix |
86+
| --- | --- | --- |
87+
| `ImportError: hamilton.async_driver` | Hamilton version too old | `pip install -U hamilton` |
88+
| `ValueError: async_driver=False` | Explicit opt-out for async runs | Use `pm.run(...)` for sync or set `async_driver=True` |
89+
| Event loop errors | Running `asyncio.run` inside an active loop | Use a framework’s lifecycle (e.g., FastAPI startup) or `nest_asyncio` in notebooks |
90+
91+
## Related
92+
93+
- [Compose Pipelines With Additional Modules](./additional-modules.md)
94+
- README: “Asynchronous Execution” section
95+

‎docs/docs/index.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ FlowerPower streamlines complex data workflows by integrating the modularity of
1212

1313
Ready to dive in? Our **[Quickstart Guide](quickstart.md)** will walk you through installing FlowerPower and running your first pipeline in just a few minutes.
1414

15+
Looking for more? Check out these guides:
16+
17+
- [Compose Pipelines With Additional Modules](guide/additional-modules.md)
18+
- [Asynchronous Execution](guide/async-execution.md)
19+
1520
## Core Concepts
1621

1722
FlowerPower is built around a few key concepts that make it both powerful and flexible:
@@ -26,4 +31,4 @@ FlowerPower is built around a few key concepts that make it both powerful and fl
2631

2732
!!! note "A Note on Hamilton"
2833

29-
FlowerPower acts as an orchestrator, not a replacement. You will still write your pipeline logic using Hamilton's function-based syntax. FlowerPower's role is to provide a structured project environment and simplify pipeline management.
34+
FlowerPower acts as an orchestrator, not a replacement. You will still write your pipeline logic using Hamilton's function-based syntax. FlowerPower's role is to provide a structured project environment and simplify pipeline management.

‎docs/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ nav:
4141
- Examples: examples.md
4242
- Guides:
4343
- Compose Pipelines With Additional Modules: guide/additional-modules.md
44+
- Asynchronous Execution: guide/async-execution.md
4445
- Advanced Usage: advanced.md
4546
- CLI Reference: cli.md
4647
- API Reference:

openspec/changes/update-async-execution-async-driver/proposal.md renamed to openspec/changes/archive/2025-11-03-update-async-execution-async-driver/proposal.md

File renamed without changes.

openspec/changes/update-async-execution-async-driver/specs/pipeline-execution/spec.md renamed to openspec/changes/archive/2025-11-03-update-async-execution-async-driver/specs/pipeline-execution/spec.md

File renamed without changes.

openspec/changes/update-async-execution-async-driver/tasks.md renamed to openspec/changes/archive/2025-11-03-update-async-execution-async-driver/tasks.md

File renamed without changes.

‎openspec/changes/docs-add-async-execution-usage/proposal.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
## Why
2+
FlowerPower now supports true asynchronous execution via Hamilton’s async driver (`hamilton.async_driver`). The documentation in `docs/` needs a clear, discoverable guide that explains how to run pipelines asynchronously, what the requirements are, and how this differs from synchronous execution.
3+
4+
## What Changes
5+
- Add a new guide page focused on async execution (proposed: `docs/docs/guide/async-execution.md`).
6+
- Explain the `PipelineManager.run_async(...)` API, the `RunConfig.async_driver` toggle, and behaviour parity with sync (`reload`, adapters, logging).
7+
- Show working examples with and without `additional_modules`.
8+
- Document prerequisites: Hamilton version providing `hamilton.async_driver`.
9+
- Troubleshooting section for common issues (missing async driver, event loop usage, executor considerations).
10+
- Update navigation in `docs/mkdocs.yml` under “Guides” to include the new page.
11+
- Cross-link from existing pages where appropriate (e.g., Advanced Usage) to the new guide.
12+
- Ensure CHANGELOG references the new async capability (already added; verify).
13+
14+
## Impact
15+
- Files to update or add:
16+
- Add: `docs/docs/guide/async-execution.md` (new page)
17+
- Update: `docs/mkdocs.yml` (nav entry)
18+
- Optional updates: link from `docs/docs/advanced.md` or `docs/docs/index.md` to the new guide
19+
20+
## Risks / Considerations
21+
- Avoid implying CLI parity if CLI subcommands do not expose async-specific flags.
22+
- Clarify that `pipelines` code must be compatible with async contexts; blocking IO will block the event loop unless executed via remote executors.
23+
- Note Hamilton version requirement; provide upgrade guidance.
24+
25+
## Acceptance Criteria
26+
- A dedicated async execution guide exists in `docs/` with code examples using `PipelineManager.run_async`.
27+
- The new guide appears in the site navigation.
28+
- The guide links to the additional-modules documentation where relevant, and vice versa when useful.
29+
- Local docs build (`mkdocs build -f docs/mkdocs.yml`) succeeds without new broken links.
30+

‎openspec/changes/docs-add-async-execution-usage/tasks.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
## 1. Documentation Updates
2+
- [x] 1.1 Create `docs/docs/guide/async-execution.md` with:
3+
- Overview of async execution and when to use it
4+
- Example using `PipelineManager.run_async` (with `additional_modules` example)
5+
- Notes on `RunConfig.async_driver` semantics (default behaviour, opting out)
6+
- Parity with sync: logging, reload, adapters
7+
- Prerequisites: Hamilton version with `hamilton.async_driver`
8+
- Troubleshooting (missing driver ImportError, event loop tips)
9+
- [x] 1.2 Update `docs/mkdocs.yml` to add the new guide to nav under Guides.
10+
- [x] 1.3 (Optional) Add cross-links in `docs/docs/advanced.md` or `docs/docs/index.md` to the new guide.
11+
- [x] 1.4 Verify CHANGELOG entry for async execution exists.
12+
13+
## 2. Validation
14+
- [x] 2.1 Build the docs locally: `mkdocs build -f docs/mkdocs.yml`.
15+
- [ ] 2.2 Fix any new warnings or broken links. *(Existing warnings predate this change and will be handled separately.)*
16+
17+
## 3. Rollout
18+
- [ ] 3.1 Submit PR for review.
19+
- [ ] 3.2 After merge/deploy, archive this change with `openspec archive docs-add-async-execution-usage --skip-specs --yes`.

‎openspec/specs/pipeline-execution/spec.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,32 @@ The system SHALL support executing a pipeline while loading additional Python mo
3737
- WHEN `final_vars` references nodes defined in any loaded module
3838
- THEN the system resolves and returns those nodes successfully
3939

40+
### Requirement: Asynchronous Execution via Hamilton Async Driver
41+
The system SHALL execute pipelines asynchronously using Hamilton’s async driver under `hamilton.async_driver`.
42+
43+
#### Scenario: Async execution succeeds with async driver
44+
- WHEN a user calls `PipelineManager.run_async("pipeline")`
45+
- THEN the system constructs the DAG with any provided `additional_modules`
46+
- AND uses `hamilton.async_driver` to execute the pipeline asynchronously
47+
- AND returns a result mapping for the requested `final_vars`
48+
49+
#### Scenario: Configure async via RunConfig
50+
- WHEN a user sets `RunConfig.async_driver=True` (or omits it) and calls `run_async`
51+
- THEN the system uses the async driver
52+
- WHEN a user sets `RunConfig.async_driver=False`
53+
- THEN the system DOES NOT use the async driver and raises a clear error or falls back per design (to be specified in design/implementation notes)
54+
55+
#### Scenario: Reload + logging parity
56+
- GIVEN `reload=True` and `log_level="DEBUG"`
57+
- WHEN a user calls `run_async`
58+
- THEN the system reloads the main and additional modules before execution
59+
- AND configures logging level for the run
60+
61+
#### Scenario: Helpful error on missing dependency
62+
- WHEN `hamilton.async_driver` cannot be imported
63+
- THEN the system raises an ImportError explaining that async execution requires a Hamilton version that provides the async driver and how to upgrade
64+
65+
#### Scenario: Adapters propagate to async
66+
- WHEN a user enables adapters via `with_adapter` and/or adapter configs
67+
- THEN the system instantiates and passes the same adapters to the async driver as in synchronous execution
68+

0 commit comments

Comments
 (0)