Allow FlowDecorators, FlowMutators and StepMutators to add_to_package#3082
Allow FlowDecorators, FlowMutators and StepMutators to add_to_package#3082romain-intel wants to merge 5 commits intomasterfrom
Conversation
Greptile SummaryThis PR extends Confidence Score: 5/5Safe to merge — only P2 style/hygiene findings, no correctness issues. All core logic is sound: USER_CONTENT dedup is handled correctly across _check_tuple/_add_tuple/_user_code_tuples, StepMutator dedup uses id() safely while objects are live, and the phase-init move in cli.py is idempotent thanks to the _ran_init guard. Only two P2 issues found: an incomplete debug.py comment and package_suffixes_mutator omitting exclude_tl_dirs from its walk() call. metaflow/plugins/package_suffixes_mutator.py — walk() call omits exclude_tl_dirs Important Files Changed
Reviews (7): Last reviewed commit: "Address comment" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
romain-intel
force-pushed
the
romain/worktree/add_files
branch
2 times, most recently
from
0612694 to
bb7d386
Compare
|
Add a PR on top to implement the package suffixes addition using this and flowmutator. |
romain-intel
force-pushed
the
romain/worktree/add_files
branch
from
bb7d386 to
311e744
Compare
| # Initialize the phase early so it can be used in the mutators | ||
| # The phase is determined by which CLI subcommand is being invoked (e.g. "run" → LAUNCH, | ||
| # "step" → TASK, "batch" → TRAMPOLINE). | ||
| system_context._update(phase=_phase_from_cli_args(getattr(ctx, "saved_args", None))) |
There was a problem hiding this comment.
what would this change when you moved it up?
There was a problem hiding this comment.
You can now use the phase in the external_init of mutators (for example)
This provides an easier way to build things that can modify what gets packaged with the .tar.gz that gets sent remotely to execute
USER_CONTENT tuples produced by FlowDecorator/StepDecorator/ FlowMutator/StepMutator add_to_package hooks were previously dropped on the floor. They are now recorded per-arcname and merged into the packaged user code: files already yielded by the flow-directory walker are skipped (dedup), and anything outside the flow dir or filtered out by the suffix/user filter still makes it into the package. Also add an example `@package_suffixes` FlowMutator that declares additional file suffixes to include in the package — mirroring the `--package-suffixes` CLI option as a decorator and demonstrating the new USER_CONTENT merging behavior.
romain-intel
force-pushed
the
romain/worktree/add_files
branch
from
66d46f0 to
2ef9783
Compare
2 participants
This provides an easier way to build things that can modify what gets packaged with the .tar.gz that gets sent remotely to execute
PR Type