Skip to content

Commit e8240e5

Browse files
authored
add docs for uv (#146)
1 parent 5d40973 commit e8240e5

File tree

6 files changed

+81
-13
lines changed

6 files changed

+81
-13
lines changed

docs/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,13 @@ Metaflow makes it easy to build and manage real-life data science, AI, and ML pr
3333
- [Managing Flows in Notebooks and Scripts](metaflow/managing-flows/introduction)
3434
- [Debugging Flows](metaflow/debugging)
3535
- [Visualizing Results](metaflow/visualizing-results/)
36-
- [Configuring Flows](metaflow/configuring-flows/introduction)*New*
36+
- [Configuring Flows](metaflow/configuring-flows/introduction)
3737

3838
## II. Scaling Flows
3939

4040
- [Introduction to Scalable Compute and Data](scaling/introduction)
4141
- [Computing at Scale](scaling/remote-tasks/introduction)
42-
- [Managing Dependencies](scaling/dependencies)
42+
- [Managing Dependencies](scaling/dependencies) *New support for `uv`*
4343
- [Dealing with Failures](scaling/failures)
4444
- [Checkpointing Progress](scaling/checkpoint/introduction)*New*
4545
- [Loading and Storing Data](scaling/data)

docs/scaling/dependencies/README.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,10 +7,15 @@ If you are in a hurry:
77
- Want to use your own modules and packages in a flow?
88
See [Structuring Projects](/scaling/dependencies/project-structure).
99
- Want to use a Python library in a flow? See [Managing Libraries](/scaling/dependencies/libraries).
10+
- Want to use `uv` with Metaflow? See [using `uv`](/scaling/dependencies/uv)
1011
- Want to use or build a specific Docker image?
1112
See [Defining Custom Containers](/scaling/dependencies/containers).
1213
:::
1314

15+
:::info
16+
New in Metaflow 2.15.8: You can [use `uv` to manage dependencies](/scaling/dependencies/uv).
17+
:::
18+
1419
A Metaflow flow is defined in a single Python file. Besides the flow code
1520
itself, a typical flow has many *software dependencies*, that is, other
1621
[modules](https://docs.python.org/3/tutorial/modules.html),
@@ -98,9 +103,10 @@ python myflow.py package list
98103
By the default, the package only includes local Python files, not libraries you
99104
have installed e.g. with `pip install` manually. To include external libraries,
100105
you need to include them either in [a custom Docker
101-
image](/scaling/dependencies/containers) or [specify them in a `@pypi` or `@conda`
102-
decorator](/scaling/dependencies/conda-vs-pypi). This figure illustrates the
103-
idea:
106+
image](/scaling/dependencies/containers), [specify them in a `@pypi` or `@conda`
107+
decorator](/scaling/dependencies/conda-vs-pypi), or [use `uv`](/scaling/dependencies/uv).
108+
109+
This figure illustrates the idea:
104110

105111
![Remote dependencies](/assets/dependencies-remote.png)
106112

docs/scaling/dependencies/conda-vs-pypi.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Let's start by outlining the two major components of the package ecosystem:
1919

2020
2. **A package manager** - a tool that downloads packages from a
2121
repository and installs them locally. The tools are often specific
22-
to a repository. For instance, `pip`, `poetry` and `pipenv` work
22+
to a repository. For instance, `uv`, `pip`, `poetry` and `pipenv` work
2323
with PyPI and `mamba` and `conda` with Conda.
2424

2525
Much confusion arises from the fact that there are many *package managers*,
@@ -32,6 +32,9 @@ Metaflow works with both the *repositories* through [the
3232
`@pypi` and `@conda`](/scaling/dependencies/libraries) decorators.
3333
Importantly, when using these decorators, you don't need to use
3434
*package managers* manually as Metaflow acts as a package manager by itself.
35+
Alternatively, an increasily popular approach in the Python ecosystem is
36+
to use `uv` as a package and project manager - and [Metaflow
37+
integrates with this approach natively](/scaling/dependencies/uv).
3538

3639
## Virtual environments
3740

@@ -43,7 +46,7 @@ are a concept closely related to package managers. In
4346
environment.
4447

4548
Some package managers like `pip` don't manage virtual environments but they rely
46-
on another tool, such as `venv` or `pyenv`, to do the job. Others like
49+
on another tool, such as `uv`, `venv` or `pyenv`, to do the job. Others like
4750
`poetry` and `mamba` handle virtual environments natively, allowing you to
4851
create and delete environments as needed.
4952

docs/scaling/dependencies/libraries.md

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,21 @@ packages](/scaling/dependencies/project-structure), this page covers handling
66
of 3rd party dependencies that are published as installable Python packages.
77

88
Metaflow supports installation of external packages from two Python package
9-
repositories: [PyPI](https://pypi.org/) and [Conda](https://conda-forge.org/),
10-
using `@pypi` and `@conda` decorators. If you wonder why we need two
11-
decorators and when to use which, see
12-
[Conda vs. PyPI](/scaling/dependencies/conda-vs-pypi). To learn why you
13-
should use the decorators and not install packages
14-
manually, see [Packaging Internals](/scaling/dependencies/internals).
9+
repositories: [PyPI](https://pypi.org/) and [Conda](https://conda-forge.org/) - learn
10+
more about their relative benefits in [Conda vs. PyPI](/scaling/dependencies/conda-vs-pypi).
11+
You can access these package repositories using `@pypi` and `@conda`
12+
decorators, or you can use [the `uv` package manager](https://github.com/astral-sh/uv)
13+
to manage PyPI dependencies in a Metaflow project similar to any other Python project.
14+
15+
Currently, the `@pypi` and `@conda` decorators provide the strongest guarantees
16+
on stability, scalability, and reproducibility of environments, as they snapshot
17+
all dependencies in the object store, as described in
18+
[Packaging Internals](/scaling/dependencies/internals). Also, they allow
19+
each step to specify a separate set of dependencies. However, you can get started
20+
easily with `uv` - in particular if you are already using it for other Python projects.
21+
22+
This page focuses on the Metaflow's `@pypi` and `@conda` decorators. If you want to
23+
use `uv` instead, see [using `uv`](/scaling/dependencies/uv).
1524

1625
## The `@pypi` and `@conda` decorators
1726

docs/scaling/dependencies/uv.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
2+
# Using uv
3+
4+
[`uv` is a fast, modern Python package manager](https://github.com/astral-sh/uv) designed for creating reproducible environments with minimal overhead. In Metaflow, you can use projects managed by `uv` directly by
5+
specifying `--environment=uv`, enabling you to manage dependencies the same way you would in any contemporary Python project. This makes it easy to develop locally and seamlessly run flows in the cloud with all dependencies included.
6+
7+
`uv` takes a more holistic approach to package and project management in contrast to `conda` and `pip`. As a result,
8+
when using `--environment=uv`, Metaflow lets `uv` install and handle dependencies using its native approach.
9+
A downside of this is that `uv` doesn't benefit from [Metaflow's internal package snapshotting](/scaling/dependencies/internals) which provides stronger guarantees for scalability and availability of packages, which matters especially
10+
when running tasks at large scale.
11+
12+
:::tip
13+
`uv` is often a convenient starting point. If you run into any issues at scale, you can always switch to `@pypi`
14+
or `@conda` later.
15+
:::
16+
17+
## Setting up a Metaflow project with `uv`
18+
19+
Metaflow works with `uv` as any other Python library, so you can use `uv` as usual. Here's an example:
20+
21+
1. Initialize a project: `uv init myflow; cd myflow`.
22+
23+
2. Add packages you need, including Metaflow: `uv add pandas metaflow`.
24+
25+
3. Run a flow `uv run flow.py run`.
26+
27+
## Running tasks remotely
28+
29+
Metaflow will package and ship the `uv` project in the cloud when you specify `--environment=uv`:
30+
31+
```
32+
uv run flow.py --environment=uv run
33+
```
34+
Or, more conveniently, you can set an environment variable
35+
```
36+
export METAFLOW_ENVIRONMENT=uv
37+
```
38+
so you don't need to add the command line option in every execution.
39+
40+
After this, you can use Metaflow's compute layer to [run the flow in the cloud](/scaling/remote-tasks/requesting-resources) with all the packages defined in your `uv` environment included, e.g. [using Kubernetes](/scaling/remote-tasks/kubernetes) ([or `--with batch`](/scaling/remote-tasks/aws-batch)):
41+
42+
```
43+
uv run flow.py run --with kubernetes
44+
```
45+
46+
Remember to set the `environment=uv` when [deploying to production](/production/introduction) as well.
47+
48+

sidebars.js

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ const sidebars = {
163163
items: [
164164
"scaling/dependencies/project-structure",
165165
"scaling/dependencies/libraries",
166+
"scaling/dependencies/uv",
166167
"scaling/dependencies/conda-vs-pypi",
167168
"scaling/dependencies/containers",
168169
"scaling/dependencies/internals",
@@ -257,6 +258,7 @@ const sidebars = {
257258
"api/step-decorators/card",
258259
"api/step-decorators/catch",
259260
"api/step-decorators/conda",
261+
"api/step-decorators/pypi",
260262
"api/step-decorators/kubernetes",
261263
"api/step-decorators/resources",
262264
"api/step-decorators/retry",

0 commit comments

Comments
 (0)