Skip to content

Commit

Permalink
modernize readme (#2318)
Browse files Browse the repository at this point in the history
last time we edited this was when we open-sourced the project. feedback
welcome!
  • Loading branch information
savingoyal authored Mar 6, 2025
1 parent 574c142 commit 1c7ef89
Show file tree
Hide file tree
Showing 2 changed files with 96 additions and 31 deletions.
72 changes: 72 additions & 0 deletions ADOPTERS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Adopters

Below is a partial list of organizations using Metaflow in production. If you'd like to be included in this list, please raise a pull request

- [23andMe](https://www.23andme.com)
- [Adept](https://www.adept.ai)
- [Amazon](https://www.amazon.com)
- [Amazon Prime Video](https://www.primevideo.com)
- [Attentive](https://www.attentive.com)
- [Autodesk](https://www.autodesk.com)
- [Bosch](https://www.bosch.com)
- [Boston Consulting Group](https://www.bcg.com)
- [Carsales](https://www.carsales.com.au)
- [Carta](https://carta.com)
- [Chess.com](https://www.chess.com)
- [CloudKitchens](https://www.cloudkitchens.com)
- [Coveo](https://www.coveo.com)
- [Crexi](https://www.crexi.com)
- [Dell](https://www.dell.com)
- [Deliveroo](https://deliveroo.com)
- [DeliveryHero](https://deliveryhero.com)
- [Disney](https://disney.com)
- [Doordash](https://doordash.com)
- [DraftKings](https://www.draftkings.com)
- [DTN](https://www.dtn.com)
- [DuckDuckGo](https://www.duckduckgo.com)
- [Dyson](https://www.dyson.com)
- [Equilibrium Energy](https://www.equilibriumenergy.com)
- [Forward Financing](https://www.forwardfinancing.com)
- [Fortum](https://www.fortum.com)
- [Genesys](https://www.genesys.com)
- [Goldman Sachs](https://www.goldmansachs.com)
- [Gradle](https://www.gradle.com)
- [GSK](https://www.gsk.com)
- [Intel](https://www.intel.com)
- [Intuitive Surgical](https://www.intuitivesurgical.com)
- [JPMorgan Chase](https://www.jpmorganchase.com)
- [Lightricks](https://www.lightricks.com)
- [Medtronic](https://www.medtronic.com)
- [Merck](https://www.merck.com)
- [Morningstar](https://www.morningstar.com)
- [Mozilla](https://www.mozilla.org)
- [Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9)
- [Nextdoor](https://www.nextdoor.com)
- [Porsche](https://www.porsche.com)
- [Pratilipi](https://www.pratilipi.com)
- [Rad.ai](https://www.rad.ai)
- [Ramp](https://ramp.com)
- [Realtor](https://www.realtor.com)
- [Roku](https://www.roku.com)
- [S&P Global](https://www.spglobal.com)
- [Sainsbury's](https://www.sainsburys.co.uk)
- [Salk Institute](https://www.salk.edu)
- [Sanofi](https://www.sanofi.com)
- [SAP](https://www.sap.com)
- [SEEK](https://www.seek.com.au)
- [Shutterstock](https://www.shutterstock.com)
- [Stanford](https://www.stanford.edu)
- [Thoughtworks](https://www.thoughtworks.com)
- [Too Good To Go](https://www.toogoodtogo.com)
- [Toyota](https://www.toyota.com)
- [Upstart](https://www.upstart.com)
- [Veriff](https://www.veriff.com)
- [Verisk](https://www.verisk.com)
- [Vouch Insurance](https://www.vouchinsurance.com)
- [Wadhwani AI](https://www.wadhwani.ai)
- [Warner Media](https://www.warnermedia.com)
- [Workiva](https://www.workiva.com)
- [Zendesk](https://www.zendesk.com)
- [Zillow](https://www.zillow.com)
- [Zipline](https://www.zipline.com)
- [Zynga](https://www.zynga.com)
55 changes: 24 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,49 @@

# Metaflow

Metaflow is a human-friendly library that helps scientists and engineers build and manage real-life data science projects. Metaflow was [originally developed at Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9) to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.
[Metaflow](https://metaflow.org) is a human-centric framework designed to help scientists and engineers **build and manage real-life AI and ML systems**. Serving teams of all sizes and scale, Metaflow streamlines the entire lifecycle, allowing rapid iteration from prototyping in notebooks to robust, maintainable production deployments — making it easier than ever to deliver robust systems.

For more information, see [Metaflow's website](https://metaflow.org) and [documentation](https://docs.metaflow.org).
Originally developed at [Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9), Metaflow is designed to boost the productivity for research and engineering teams working on [a wide variety of projects](https://netflixtechblog.com/supporting-diverse-ml-systems-at-netflix-2d2e6b6d205d), from classical statistics to state-of-the-art deep learning and foundation models. Metaflow helps unify code, data, and compute at every stage, ensuring robust, end-to-end systems for real-world AI and ML.

Today, Metaflow powers thousands of AI and ML experiences across a diverse array of companies, large and small, including Amazon, Doordash, Dyson, Goldman Sachs, Ramp, and [more](ADOPTERS.md). At Netflix, Metaflow supports over 3000 AI and ML systems, executing hundreds of millions of data-intensive, high-performance compute jobs and managing tens of petabytes of models and artifacts in it's datastore across hundreds of users within Netflix's AI, ML, data science, and engineering teams.

## From prototype to production (and back)

Metaflow provides a simple, friendly API that covers foundational needs of ML, AI, and data science projects:
Metaflow provides a simple and friendly pythonic [API](https://docs.metaflow.org) that covers foundational needs of AI and ML systems:
<img src="./docs/prototype-to-prod.png" width="800px">

1. [Rapid local prototyping](https://docs.metaflow.org/metaflow/basics), [support for notebooks](https://docs.metaflow.org/metaflow/visualizing-results), and [built-in experiment tracking and versioning](https://docs.metaflow.org/metaflow/client).
2. [Horizontal and vertical scalability to the cloud](https://docs.metaflow.org/scaling/remote-tasks/introduction), utilizing both CPUs and GPUs, and [fast data access](https://docs.metaflow.org/scaling/data).
3. [Managing dependencies](https://docs.metaflow.org/scaling/dependencies) and [one-click deployments to highly available production orchestrators](https://docs.metaflow.org/production/introduction).
1. [Rapid local prototyping](https://docs.metaflow.org/metaflow/basics), [support for notebooks](https://docs.metaflow.org/metaflow/managing-flows/notebook-runs), and built-in support for [experiment tracking, versioning](https://docs.metaflow.org/metaflow/client) and [visualization](https://docs.metaflow.org/metaflow/visualizing-results).
2. [Effortlessly scale horizontally and vertically in your cloud](https://docs.metaflow.org/scaling/remote-tasks/introduction), utilizing both CPUs and GPUs, with [fast data access](https://docs.metaflow.org/scaling/data) for running [massive embarrassingly parallel](https://docs.metaflow.org/metaflow/basics#foreach) as well as [gang-scheduled](https://docs.metaflow.org/scaling/remote-tasks/distributed-computing) compute workloads [reliably](https://docs.metaflow.org/scaling/failures) and [efficiently](https://docs.metaflow.org/scaling/checkpoint/introduction).
3. [Easily manage dependencies](https://docs.metaflow.org/scaling/dependencies) and [deploy with one-click](https://docs.metaflow.org/production/introduction) to highly available production orchestrators with built in support for [reactive orchestration](https://docs.metaflow.org/production/event-triggering).

For full documentation, check out our [API Reference](https://docs.metaflow.org/api) or see our [Release Notes](https://github.com/Netflix/metaflow/releases) for the latest features and improvements.


## Getting started

Getting up and running is easy. If you don't know where to start, [Metaflow sandbox](https://outerbounds.com/sandbox) will have you running and exploring Metaflow in seconds.
Getting up and running is easy. If you don't know where to start, [Metaflow sandbox](https://outerbounds.com/sandbox) will have you running and exploring in seconds.

### Installing Metaflow in your Python environment
### Installing Metaflow

To install Metaflow in your local environment, you can install from [PyPi](https://pypi.org/project/metaflow/):
To install Metaflow in your Python environment from [PyPI](https://pypi.org/project/metaflow/):

```sh
pip install metaflow
```
Alternatively, you can also install from [conda-forge](https://anaconda.org/conda-forge/metaflow):
Alternatively, using [conda-forge](https://anaconda.org/conda-forge/metaflow):

```sh
conda install -c conda-forge metaflow
```
If you are eager to try out Metaflow in practice, you can start with the [tutorial](https://docs.metaflow.org/getting-started/tutorials). After the tutorial, you can learn more about how Metaflow works [here](https://docs.metaflow.org/metaflow/basics).

Once installed, a great way to get started is by following our [tutorial](https://docs.metaflow.org/getting-started/tutorials). It walks you through creating and running your first Metaflow flow step by step.

For more details on Metaflow’s features and best practices, check out:
- [How Metaflow works](https://docs.metaflow.org/metaflow/basics)
- [Additional resources](https://docs.metaflow.org/introduction/metaflow-resources)

If you need help, don’t hesitate to reach out on our [Slack community](http://slack.outerbounds.co/)!


### Deploying infrastructure for Metaflow in your cloud
<img src="./docs/multicloud.png" width="800px">
Expand All @@ -42,28 +54,9 @@ While you can get started with Metaflow easily on your laptop, the main benefits
and to [deploy to production-grade workflow orchestrators](https://docs.metaflow.org/production/introduction). To benefit from these features, follow this [guide](https://outerbounds.com/engineering/welcome/) to
configure Metaflow and the infrastructure behind it appropriately.

## [Resources](https://docs.metaflow.org/introduction/metaflow-resources)

### [Slack Community](http://slack.outerbounds.co/)
An active [community](http://slack.outerbounds.co/) of thousands of data scientists and ML engineers discussing the ins-and-outs of applied machine learning.

### [Tutorials](https://outerbounds.com/docs/tutorials-index/)
- [Introduction to Metaflow](https://outerbounds.com/docs/intro-tutorial-overview/)
- [Natural Language Processing with Metaflow](https://outerbounds.com/docs/nlp-tutorial-overview/)
- [Computer Vision with Metaflow](https://outerbounds.com/docs/cv-tutorial-overview/)
- [Recommender Systems with Metaflow](https://outerbounds.com/docs/recsys-tutorial-overview/)
- And more advanced content [here](https://outerbounds.com/docs/tutorials-index/)

### [Generative AI and LLM use cases](https://outerbounds.com/blog/?category=Foundation%20Models)
- [Infrastructure Stack for Large Language Models](https://outerbounds.com/blog/llm-infrastructure-stack/)
- [Parallelizing Stable Diffusion for Production Use Cases](https://outerbounds.com/blog/parallelizing-stable-diffusion-production-use-cases/)
- [Whisper with Metaflow on Kubernetes](https://outerbounds.com/blog/whisper-kubernetes/)
- [Training a Large Language Model With Metaflow, Featuring Dolly](https://outerbounds.com/blog/train-dolly-metaflow/)

## Get in touch
There are several ways to get in touch with us:
- [Slack Community](http://slack.outerbounds.co/)
- [Github Issues](https://github.com/Netflix/metaflow/issues)
We'd love to hear from you. Join our community [Slack workspace]((http://slack.outerbounds.co/))!

## Contributing
We welcome contributions to Metaflow. Please see our [contribution guide](https://docs.metaflow.org/introduction/contributing-to-metaflow) for more details.
Expand Down

0 comments on commit 1c7ef89

Please sign in to comment.