Commit 8a12f03
## PR Type
- [ ] Bug fix
- [X] New feature
- [X] Core Runtime change (higher bar -- see
[CONTRIBUTING.md](../CONTRIBUTING.md#core-runtime-contributions-higher-bar))
- [ ] Docs / tooling
- [ ] Refactoring
## Summary
Makes the FlowSpec graph more flexible in three ways, with backward
compatibility preserved throughout:
1. **Explicit start/end annotations.** `@step` now accepts `start=True`
and `end=True` kwargs. Steps no longer have to be named "start"/"end" —
a flow can use any name as long as one step is annotated as the start
and one as the end. When no annotations are present, the graph falls
back to discovering steps by the names "start" and "end" (old behavior).
2. **Single-step flows.** A single step annotated `@step(start=True,
end=True)` is now valid; previously a FlowSpec required at least two
steps.
3. **Per-node `node_info` metadata.** `DAGNode` gains a `node_info` dict
that extensions can populate to attach arbitrary per-step metadata. Live
references are accessible via `flow._graph`; serialized values (via
`to_pod`) flow into `_graph_info`. Callables are serialized as their
qualified name.
These are core-runtime changes because they touch `graph.py`, `lint.py`,
`runtime.py`, `task.py`, and every orchestrator plugin (Argo Workflows,
Step Functions, Airflow) that reads the graph or emits events tied to
the terminal step.
### Key changes
| Area | What changed |
|------|-------------|
| `decorators.py` | `@step(start=False, end=False, node_info=None)` |
| `graph.py` | `DAGNode.is_start_step`, `is_end_step`, `node_info`;
`_identify_start_end` uses annotations with name fallback; attribute
name (not AST `def` name) is authoritative; tolerates sourceless
single-step methods (for out-of-tree FunctionSpec extension) |
| `lint.py` | `check_basic_steps` distinguishes "no start" from
"multiple start annotations"; new `check_start_end_degree` validates
in/out degrees; new `check_annotation_name_conflict` warns when
`@step(start=True)` coexists with a step named "start" (same for end) |
| `util.py` | `to_pod` serializes callables via `__qualname__` |
| `runtime.py` | Persists `start_step`/`end_step` as metadata on the
`_parameters` task |
| `task.py`, `flowspec.py` | Use the graph's start/end step rather than
literal "start"/"end" names |
| `client/core.py` | `Run._graph_endpoints` reads endpoints from
`_parameters` metadata with `_graph_info` fallback for orchestrated runs
|
| `plugins/argo/argo_workflows.py` | `_matching_conditional_join` falls
back to `graph.end_step`; terminal-step detection through the DAG |
| `plugins/argo/argo_workflows_decorator.py` | Auto-emitted event from
the end step is also published with the well-known `.end` suffix so
`@trigger_on_finish` subscribers — which don't know the publisher's end
step name at deploy time — keep receiving triggers from custom-named end
steps |
| `plugins/aws/step_functions/step_functions.py` | Reads `StartAt` from
the definition JSON instead of hardcoded `["States"]["start"]` |
| `plugins/airflow/airflow.py`, `airflow_cli.py` | Use
`graph.start_step`/`end_step` instead of hardcoded names |
| `plugins/cards/card_cli.py` | `graph` kwarg keeps the old
dict-of-steps shape for backward compatibility with third-party card
modules; new opt-in `graph_info` kwarg carries the full payload (steps +
start/end). Card_cli probes the constructor signature and only passes
`graph_info` to cards that accept it |
| `plugins/cards/card_modules/basic.py` | `DefaultCard`,
`DefaultCardJSON`, `ErrorCard` accept both `graph` and `graph_info`
(preferred when provided) |
| `plugins/cards/ui/` | Svelte DAG renderer reads start/end dynamically
instead of assuming "start"/"end" |
## Tests
- [X] Unit tests added/updated
- [ ] Reproduction script provided (required for Core Runtime)
- [X] CI passes
- [ ] If tests are impractical: explain why below and provide manual
evidence above
New and updated tests cover:
- Graph structure inference: standard flows, custom-named flows,
single-step flows, branch flows, foreach flows, split-as-entry-step
flows
- `@step(start=True)`/`@step(end=True)` annotation mechanics and
backward-compat with name-based discovery
- Lint validation: degree checks, annotation-name conflicts,
negative-path cases for malformed annotations
- End-to-end execution of custom-named flows via Runner with card
rendering
- `Trigger.from_runs` on flows with a custom-named terminal step
- Step Functions `StartAt` lookup from deployment JSON
- Card graph transform accepting both legacy and new payload shapes
## Non-Goals
- A `StepSpec`/`FunctionSpec` construct (single-step FlowSpec-like class
with `init()`+`call()` lifecycle) — hooks for it are present in
`DAGNode` and `_base_step_decorator` (see module comment in `graph.py`),
but the class itself is kept out-of-tree for now.
- Multi-flow file support (a `FlowSpec.main()` classmethod to route
between multiple FlowSpec subclasses in one file) — deferred.
- Direct non-CLI invocation of FlowSpec — deferred.
## AI Tool Usage
- [X] AI tools were used (describe below)
AI coding assistance was used during development. All generated code was
reviewed and tested.
---------
Co-authored-by: Shashank Srikanth <ssrikanth@netflix.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 73db930 commit 8a12f03
44 files changed
Lines changed: 2294 additions & 210 deletions
File tree
- metaflow
- client
- plugins
- airflow
- argo
- aws/step_functions
- cards
- card_modules
- ui/src
- components/dag
- test
- core/tests
- unit
- graph_inference
- flows
- localbatch
- ux/core
- flows/basic
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2121 | 2121 | | |
2122 | 2122 | | |
2123 | 2123 | | |
| 2124 | + | |
2124 | 2125 | | |
2125 | | - | |
| 2126 | + | |
2126 | 2127 | | |
2127 | 2128 | | |
2128 | 2129 | | |
| |||
2139 | 2140 | | |
2140 | 2141 | | |
2141 | 2142 | | |
| 2143 | + | |
2142 | 2144 | | |
2143 | | - | |
| 2145 | + | |
2144 | 2146 | | |
2145 | 2147 | | |
2146 | 2148 | | |
| |||
2153 | 2155 | | |
2154 | 2156 | | |
2155 | 2157 | | |
2156 | | - | |
| 2158 | + | |
2157 | 2159 | | |
2158 | 2160 | | |
2159 | 2161 | | |
| |||
2165 | 2167 | | |
2166 | 2168 | | |
2167 | 2169 | | |
2168 | | - | |
| 2170 | + | |
2169 | 2171 | | |
2170 | 2172 | | |
2171 | 2173 | | |
| |||
2176 | 2178 | | |
2177 | 2179 | | |
2178 | 2180 | | |
| 2181 | + | |
| 2182 | + | |
| 2183 | + | |
| 2184 | + | |
| 2185 | + | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
| 2195 | + | |
| 2196 | + | |
| 2197 | + | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
2179 | 2211 | | |
2180 | 2212 | | |
2181 | 2213 | | |
| |||
2298 | 2330 | | |
2299 | 2331 | | |
2300 | 2332 | | |
2301 | | - | |
| 2333 | + | |
2302 | 2334 | | |
2303 | | - | |
| 2335 | + | |
2304 | 2336 | | |
2305 | 2337 | | |
2306 | 2338 | | |
2307 | 2339 | | |
2308 | | - | |
| 2340 | + | |
2309 | 2341 | | |
2310 | 2342 | | |
2311 | | - | |
| 2343 | + | |
| 2344 | + | |
2312 | 2345 | | |
2313 | 2346 | | |
2314 | 2347 | | |
| |||
2481 | 2514 | | |
2482 | 2515 | | |
2483 | 2516 | | |
2484 | | - | |
2485 | | - | |
| 2517 | + | |
| 2518 | + | |
| 2519 | + | |
2486 | 2520 | | |
2487 | 2521 | | |
2488 | 2522 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
588 | 588 | | |
589 | 589 | | |
590 | 590 | | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
591 | 612 | | |
592 | 613 | | |
593 | 614 | | |
| |||
1009 | 1030 | | |
1010 | 1031 | | |
1011 | 1032 | | |
1012 | | - | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
1013 | 1038 | | |
1014 | 1039 | | |
1015 | 1040 | | |
| |||
1034 | 1059 | | |
1035 | 1060 | | |
1036 | 1061 | | |
1037 | | - | |
1038 | | - | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
1039 | 1074 | | |
1040 | 1075 | | |
1041 | 1076 | | |
1042 | | - | |
1043 | | - | |
| 1077 | + | |
| 1078 | + | |
1044 | 1079 | | |
1045 | | - | |
1046 | | - | |
1047 | | - | |
1048 | | - | |
1049 | | - | |
1050 | | - | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
1051 | 1097 | | |
1052 | 1098 | | |
1053 | 1099 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
62 | 69 | | |
63 | 70 | | |
64 | 71 | | |
65 | | - | |
66 | | - | |
| 72 | + | |
| 73 | + | |
67 | 74 | | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
72 | 93 | | |
73 | 94 | | |
74 | 95 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
522 | 522 | | |
523 | 523 | | |
524 | 524 | | |
| 525 | + | |
| 526 | + | |
525 | 527 | | |
526 | 528 | | |
527 | 529 | | |
| |||
0 commit comments