This repository contains various configuration files, tools and utilities for operating Metaflow in production. See Metaflow documentation for more information about Metaflow architecture. Top level folders are structured as follows:
If you're not already using Terraform, this is the easiest way to get started on AWS. You can find the template under aws/cloudformation in this repository. This stack uses AWS Batch for compute and AWS Step Functions for orchestration.
Another deployment option is using Terraform. An end-to-end example can be found under aws/terraform in this repository. The example leverages the official Terraform module outerbounds/terraform-aws-metaflow as a building block. This stack uses AWS Batch for compute and AWS Step Functions for orchestration.
This is the quickest way to spin up a fully functional Metaflow stack on Azure. See details under azure/terraform in this repository. This stack uses Kubernetes (AKS) for compute and Argo Workflows for orchestration.
This is another quickest way to spin up a fully functional Metaflow stack on Nebius. See details under nebius/terraform in this repository. This stack uses Kubernetes (Managed Service for Kubernetes in Nebius AI Cloud) for compute and Argo Workflows for orchestration.
We provide Helm charts to deploy Metaflow Metadata service and UI in a K8S cluster. This way you can use Metaflow without any AWS-specific dependencies on AWS except for having a S3-compatible object storage engine available. You can find them under k8s/helm/metaflow in this repository.
Some Metaflow flows that could be used to test drive a Metaflow stack. Some of these flows are used to drive end-to-end CI coverage internally at Outerbounds. They live under common/sample_flows
Scripts that make life easier either deploying or using your new Metaflow stacks.
Talk to us on Slack.