Question on Executing Metaflow Workflow from Python Script Without 'run' Argument

[sungreong]

Hello Metaflow Team,

I am exploring ways to automate Metaflow workflows and have a query regarding the initial execution of these workflows via a Python script. Specifically, I'm interested in whether it is possible to execute a Metaflow workflow directly from a script without explicitly using the run argument for the first time.

Could you provide guidance or confirm if there's a recommended approach for initializing and running workflows programmatically without the run command? Any insights on setting up the environment or script adjustments to handle this use case would be greatly appreciated.

from metaflow import FlowSpec, step

class ExampleFlow(FlowSpec):
    @step
    def start(self):
        print("This is the start step.")
        self.next(self.end)

    @step
    def end(self):
        print("This is the end step.")

if __name__ == '__main__':
# How to initiate this flow without using 'ExampleFlow().run()'?
# HelloFlow 인스턴스 생성
flow = ExampleFlow()
graph = flow._graph
current_steps = ['start']

while current_steps:
    next_steps = []
    for current_step in current_steps:
        print(f"Running step: {current_step}")
        run_step(flow, current_step)
        next_steps.extend(step.__name__ for step in flow._next_steps)

    print("Next steps:", next_steps)
    current_steps = next_steps

Thank you for your assistance!

[romain-intel]

May I ask for your use case specifically. Not exactly what you are asking for but we will soon have a runner (ie: able to run in python) but Metaflow will handle the running itself. It should be equivalent to what you are trying to do here (unless you want to do funky things between steps). See here: #1732. This should be out soon.

[sungreong]

May I ask for your use case specifically. Not exactly what you are asking for but we will soon have a runner (ie: able to run in python) but Metaflow will handle the running itself. It should be equivalent to what you are trying to do here (unless you want to do funky things between steps). See here: #1732. This should be out soon.

Thank you for your response and the ongoing developments in Metaflow. My main objective is to simplify the execution of Metaflow workflows, transitioning from the traditional command python train.py run to simply python train.py. This adjustment would enhance the usability and integration of Metaflow within our automation scripts and larger application ecosystem.

Here is how I am currently setting up the workflow to achieve this:

from metaflow import FlowSpec, step

class ExampleFlow(FlowSpec):
    @step
    def start(self):
        print("This is the start step.")
        self.next(self.end)

    @step
    def end(self):
        print("This is the end step.")

if __name__ == '__main__':
    ExampleFlow().run()

By including ExampleFlow().run() in the if name == 'main': block, the workflow initiates directly when the script is executed with python train.py. This setup works for our basic workflows, and I am curious to know if this method aligns with best practices within the Metaflow framework.

[savingoyal]

@sungreong would you be able to use the functionality called out in this blog - https://outerbounds.com/blog/metaflow-in-notebooks-and-scripts

[savingoyal]

@sungreong, please feel free to reopen this issue if you have further questions or comments