Skip to content

Commit 4e5e736

Browse files
benpankowbraunjj
benpankow
authored and
braunjj
committed
[components docs] Flesh out existing code location docs, test (#27604)
## Summary Adds a section to the 'existing project' docs about adding to `pyproject.toml` (necessary for `dg` cli to work) alongside optionally configuring a `uv` environment. This was after playing around trying to port a DOP project, I found some of these steps not immediately obvious & had a hard time setting up a new venv. Places the existing + new contents under integration test.
1 parent 83442d3 commit 4e5e736

25 files changed

+323
-31
lines changed

‎docs/docs-beta/docs/guides/preview/components/existing-code-location.md

+48-9
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,50 @@ sidebar_position: 300
77
This guide is only relevant if you are starting from an _existing_ Dagster code location. This setup is unnecessary if you used `dg code-location generate` to create your code location.
88
:::
99

10-
## Create a `components` directory
10+
Let's begin with a Dagster code location that has some assets, but no components:
1111

12-
First, you'll want to create a directory to contain any new components you add to your code location. By convention, this directory is named `components`, and exists at the top level of your code location's Python module.
12+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/1-tree.txt" />
1313

14-
```bash
15-
mkdir components
16-
```
14+
## Install dependencies
15+
16+
### Install the `dg` command line tool
17+
18+
We'll need to install the `dg` command line tool, which is used to scaffold components. We recommend installing `dg` globally using the [`uv`](https://docs.astral.sh/uv/getting-started/installation/) package manager; it can also be installed using `pip`.
19+
20+
<CliInvocationExample contents="uv tool install dagster-dg" />
21+
22+
### Install `dagster-components`
23+
24+
Next, we'll need to install the `dagster-components` package.
25+
26+
Though this is optional, we generally recommend using a separate virtual environment for each code location, which can be accomplished using `uv`:
27+
28+
<Tabs>
29+
<TabItem value='before' label='Using uv virtual environment'>
30+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/3-uv-venv.txt" />
31+
32+
Then, we can use `uv sync` to install the dependencies from our `pyproject.toml`, and then install the `dagster-components` package:
33+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/4-uv-freeze.txt" />
34+
</TabItem>
35+
<TabItem value='after' label='Using pip'>
36+
<CliInvocationExample contents="pip install dagster-components" />
37+
</TabItem>
38+
</Tabs>
39+
40+
## Update project structure
41+
42+
### Update `pyproject.toml`
43+
44+
Add a `tool.dg` section to your `pyproject.toml` file. This will tell the `dg` command line tool that this code location is a valid Dagster code location.
45+
46+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/2-pyproject.toml" language="toml" title="pyproject.toml" />
47+
48+
49+
### Create a `components` directory
50+
51+
Next, you'll want to create a directory to contain any new components you add to your code location. By convention, this directory is named `components`, and exists at the top level of your code location's Python module.
52+
53+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/5-mkdir-components.txt" />
1754

1855
## Modify top-level definitions
1956

@@ -25,14 +62,16 @@ You can manually construct a set of definitions for your components using `build
2562

2663
<Tabs>
2764
<TabItem value='before' label='Before'>
28-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-before.py" language="python" />
65+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/6-initial-definitions.py" language="python" />
2966
</TabItem>
3067
<TabItem value='after' label='After'>
31-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-after.py" language="python" />
68+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/7-updated-definitions.py" language="python" />
3269
</TabItem>
3370
</Tabs>
3471

72+
Now, your code location is ready to use components! `dg` can be used to scaffold new components directly into the existing code location.
73+
3574
## Next steps
3675

37-
- Add a new component to your code location
38-
- Create a new component type
76+
- [Add a new component to your code location](./using-a-component)
77+
- [Create a new component type](./creating-a-component)

‎docs/docs-beta/src/code-examples-content.js

+9-5
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/1-tree.txt

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
tree
2+
3+
.
4+
├── README.md
5+
├── my_existing_project
6+
│   ├── __init__.py
7+
│   ├── assets.py
8+
│   └── definitions.py
9+
├── my_existing_project_tests
10+
│   ├── __init__.py
11+
│   └── test_assets.py
12+
└── pyproject.toml
13+
14+
3 directories, 7 files

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/2-pyproject.toml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
...
2+
[tool.dg]
3+
is_code_location = true
4+
5+
[tool.dagster]
6+
module_name = "my_existing_project.definitions"
7+
code_location_name = "my_existing_project"
8+
...

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/3-uv-venv.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
uv venv

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/4-uv-freeze.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
uv sync && uv add dagster-components

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/5-mkdir-components.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
mkdir my_existing_project/components

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/6-initial-definitions.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
import dagster as dg
2+
from my_existing_project import assets
3+
4+
all_assets = dg.load_assets_from_modules([assets])
5+
6+
defs = dg.Definitions(
7+
assets=all_assets,
8+
)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-after.py renamed to ‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/7-updated-definitions.py

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
from pathlib import Path
22

33
import dagster_components as dg_components
4+
from my_existing_project import assets
45

56
import dagster as dg
67

8+
all_assets = dg.load_assets_from_modules([assets])
9+
710
defs = dg.Definitions.merge(
8-
dg.Definitions(assets=[]),
11+
dg.Definitions(assets=all_assets),
912
dg_components.build_component_defs(Path(__file__).parent / "components"),
1013
)

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/8-dg-list-component-types.txt

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
dg component-type list
2+
3+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
4+
┃ Component Type ┃ Summary ┃
5+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
6+
│ definitions@dagster_components │ Wraps an arbitrary set of │
7+
│ │ Dagster definitions. │
8+
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
9+
│ │ scripts executed with │
10+
│ │ Dagster's │
11+
│ │ PipesSubprocessClient. │
12+
└───────────────────────────────────────────────────────┴────────────────────────────────┘

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-before.py

-3
This file was deleted.

‎examples/docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/pyproject.toml

-2
This file was deleted.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Sample existing project for testing docs for the "Making an existing code location components-compatible" guide.

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/assets.py

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from dagster import asset
2+
3+
4+
@asset
5+
def my_asset():
6+
pass

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project/definitions.py

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
import dagster as dg
2+
from my_existing_project import assets
3+
4+
all_assets = dg.load_assets_from_modules([assets])
5+
6+
defs = dg.Definitions(
7+
assets=all_assets,
8+
)

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project_tests/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/my_existing_project_tests/test_assets.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/my-existing-project/pyproject.toml

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
[project]
2+
name = "my_existing_project"
3+
version = "0.1.0"
4+
description = "Add your description here"
5+
readme = "README.md"
6+
requires-python = ">=3.9,<3.13"
7+
dependencies = [
8+
"dagster",
9+
]
10+
11+
[project.optional-dependencies]
12+
dev = [
13+
"dagster-webserver",
14+
"pytest>8",
15+
]
16+
17+
[build-system]
18+
requires = ["setuptools"]
19+
build-backend = "setuptools.build_meta"
20+
21+
[tool.dagster]
22+
module_name = "my_existing_project.definitions"
23+
code_location_name = "my_existing_project"
24+
25+
[tool.setuptools.packages.find]
26+
exclude=["my_existing_project_tests"]

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs.py

-1
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,6 @@ def next_snip_no():
5858
),
5959
):
6060
os.chdir(tempdir)
61-
subprocess.check_call(["uv", "pip", "install", "dg"])
6261

6362
run_command_and_snippet_output(
6463
cmd="dg --help",

‎examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/guides/components/test_components_docs_deployments.py

-1
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,6 @@ def next_snip_no():
5353
),
5454
):
5555
os.chdir(tempdir)
56-
subprocess.check_call(["uv", "pip", "install", "dg"])
5756

5857
# Scaffold deployment
5958
run_command_and_snippet_output(

0 commit comments

Comments
 (0)