Skip to content

[components docs] Flesh out existing code location docs, test #27604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,50 @@ sidebar_position: 300
This guide is only relevant if you are starting from an _existing_ Dagster code location. This setup is unnecessary if you used `dg code-location generate` to create your code location.
:::

## Create a `components` directory
Let's begin with a Dagster code location that has some assets, but no components:

First, you'll want to create a directory to contain any new components you add to your code location. By convention, this directory is named `components`, and exists at the top level of your code location's Python module.
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/1-tree.txt" />

```bash
mkdir components
```
## Install dependencies

### Install the `dg` command line tool

We'll need to install the `dg` command line tool, which is used to scaffold components. We recommend installing `dg` globally using the [`uv`](https://docs.astral.sh/uv/getting-started/installation/) package manager; it can also be installed using `pip`.

<CliInvocationExample contents="uv tool install dagster-dg" />

### Install `dagster-components`

Next, we'll need to install the `dagster-components` package.

Though this is optional, we generally recommend using a separate virtual environment for each code location, which can be accomplished using `uv`:

<Tabs>
<TabItem value='before' label='Using uv virtual environment'>
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/3-uv-venv.txt" />

Then, we can use `uv sync` to install the dependencies from our `pyproject.toml`, and then install the `dagster-components` package:
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/4-uv-freeze.txt" />
</TabItem>
<TabItem value='after' label='Using pip'>
<CliInvocationExample contents="pip install dagster-components" />
</TabItem>
</Tabs>

## Update project structure

### Update `pyproject.toml`

Add a `tool.dg` section to your `pyproject.toml` file. This will tell the `dg` command line tool that this code location is a valid Dagster code location.

<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/2-pyproject.toml" language="toml" title="pyproject.toml" />


### Create a `components` directory

Next, you'll want to create a directory to contain any new components you add to your code location. By convention, this directory is named `components`, and exists at the top level of your code location's Python module.

<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/5-mkdir-components.txt" />

## Modify top-level definitions

Expand All @@ -25,14 +62,16 @@ You can manually construct a set of definitions for your components using `build

<Tabs>
<TabItem value='before' label='Before'>
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-before.py" language="python" />
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/6-initial-definitions.py" language="python" />
</TabItem>
<TabItem value='after' label='After'>
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/definitions-after.py" language="python" />
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/existing-project/7-updated-definitions.py" language="python" />
</TabItem>
</Tabs>

Now, your code location is ready to use components! `dg` can be used to scaffold new components directly into the existing code location.

## Next steps

- Add a new component to your code location
- Create a new component type
- [Add a new component to your code location](./using-a-component)
- [Create a new component type](./creating-a-component)
14 changes: 9 additions & 5 deletions docs/docs-beta/src/code-examples-content.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
tree

.
├── README.md
├── my_existing_project
│   ├── __init__.py
│   ├── assets.py
│   └── definitions.py
├── my_existing_project_tests
│   ├── __init__.py
│   └── test_assets.py
└── pyproject.toml

3 directories, 7 files
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
...
[tool.dg]
is_code_location = true

[tool.dagster]
module_name = "my_existing_project.definitions"
code_location_name = "my_existing_project"
...
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
uv venv
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
uv sync && uv add dagster-components
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mkdir my_existing_project/components
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import dagster as dg
from my_existing_project import assets

all_assets = dg.load_assets_from_modules([assets])

defs = dg.Definitions(
assets=all_assets,
)
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
from pathlib import Path

import dagster_components as dg_components
from my_existing_project import assets

import dagster as dg

all_assets = dg.load_assets_from_modules([assets])

defs = dg.Definitions.merge(
dg.Definitions(assets=[]),
dg.Definitions(assets=all_assets),
dg_components.build_component_defs(Path(__file__).parent / "components"),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
dg component-type list

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ definitions@dagster_components │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Sample existing project for testing docs for the "Making an existing code location components-compatible" guide.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from dagster import asset


@asset
def my_asset():
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import dagster as dg
from my_existing_project import assets

all_assets = dg.load_assets_from_modules([assets])

defs = dg.Definitions(
assets=all_assets,
)
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
[project]
name = "my_existing_project"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.9,<3.13"
dependencies = [
"dagster",
]

[project.optional-dependencies]
dev = [
"dagster-webserver",
"pytest>8",
]

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.dagster]
module_name = "my_existing_project.definitions"
code_location_name = "my_existing_project"

[tool.setuptools.packages.find]
exclude=["my_existing_project_tests"]
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,6 @@ def next_snip_no():
),
):
os.chdir(tempdir)
subprocess.check_call(["uv", "pip", "install", "dg"])

run_command_and_snippet_output(
cmd="dg --help",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ def next_snip_no():
),
):
os.chdir(tempdir)
subprocess.check_call(["uv", "pip", "install", "dg"])

# Scaffold deployment
run_command_and_snippet_output(
Expand Down
Loading