Skip to content

Commit 4ab0148

Browse files
benpankowbraunjj
authored andcommitted
[dg docs] Add docs e2e test, screenshot generation for component type tutorial (#27633)
## Summary Noticed these docs are already(!) out of date (uses generate instead of scaffold verbiage, `.resolve_params` instead of `.resolve`), so added a brief test. Also played with some screenshot generation. Didn't touch the last section on advanced custom resolution because the code snippets are pretty pseudocode-y, bonus points if someone wants to put that under test. ## Test Plan See [new docs](https://dagster-docs-agnb11wv4-elementl.vercel.app/guides/preview/components/creating-a-component) vs [existing docs](https://docs.dagster.io/guides/preview/components/creating-a-component)
1 parent cff50a4 commit 4ab0148

File tree

16 files changed

+732
-17
lines changed

16 files changed

+732
-17
lines changed

docs/docs-beta/src/code-examples-content.js

+387
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/docs/guides/preview/components/creating-a-component.md

+7-12
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: 'Creating a New Component Type'
2+
title: 'Creating a new component type'
33
sidebar_position: 100
44
---
55

@@ -36,13 +36,11 @@ For this example, we'll write a lightweight component that executes a shell comm
3636

3737
First, we use the `dg` command-line utility to scaffold a new component type:
3838

39-
```bash
40-
dg component-type generate shell_command
41-
```
39+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/1-dg-scaffold-shell-command.txt" />
4240

4341
This will add a new file to your project in the `lib` directory:
4442

45-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/empty.py" language="python" />
43+
<CodeExample path="docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/2-shell-command-empty.py" language="python" title="my_component_library/lib/shell_command.py" />
4644

4745
This file contains the basic structure for the new component type. There are two methods that you'll need to implement:
4846

@@ -92,18 +90,15 @@ Our `build_defs` method will create a single `@asset` that executes the provided
9290

9391
Following the steps above will automatically register your component type in your environment. You can now run:
9492

95-
```bash
96-
dg component-type list
97-
```
93+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/3-dg-list-component-types.txt" />
9894

9995
and see your new component type in the list of available component types.
10096

10197
You can also view automatically generated documentation describing your new component type by running:
10298

103-
```bash
104-
dg component-type docs your_library.shell_command
105-
```
99+
<CliInvocationExample path="docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/4-dg-component-type-docs.txt" />
106100

101+
![](/images/guides/build/projects-and-components/components/component-type-docs.png)
107102
## [Advanced] Custom templating
108103

109104
The components system supports a rich templating syntax that allows you to load arbitrary Python values based off of your `component.yaml` file. All string values in a `ResolvableModel` can be templated using the Jinja2 templating engine, and may be resolved into arbitrary Python types. This allows you to expose complex object types, such as `PartitionsDefinition` or `AutomationCondition` to users of your component, even if they're working in pure YAML.
@@ -126,4 +121,4 @@ params:
126121
127122
## Next steps
128123
129-
- Add a new component to your project
124+
- [Add a new component to your project](./using-a-component.md)
Loading
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
dg component-type scaffold shell_command
2+
3+
Creating a Dagster component type at /.../my-component-library/my_component_library/lib/shell_command.py.
4+
Scaffolded files for Dagster project in /.../my-component-library/my_component_library/lib.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
from dagster import Definitions
2+
from dagster_components import (
3+
Component,
4+
ComponentLoadContext,
5+
DefaultComponentScaffolder,
6+
ResolvableSchema,
7+
registered_component_type,
8+
)
9+
10+
class ShellCommandSchema(ResolvableSchema):
11+
...
12+
13+
@registered_component_type(name="shell_command")
14+
class ShellCommand(Component):
15+
"""COMPONENT SUMMARY HERE.
16+
17+
COMPONENT DESCRIPTION HERE.
18+
"""
19+
20+
@classmethod
21+
def get_schema(cls):
22+
return ShellCommandSchema
23+
24+
@classmethod
25+
def get_scaffolder(cls) -> DefaultComponentScaffolder:
26+
return DefaultComponentScaffolder()
27+
28+
def build_defs(self, load_context: ComponentLoadContext) -> Definitions:
29+
# Add definition construction logic here.
30+
return Definitions()
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
dg component-type list
2+
3+
Using /.../my-component-library/.venv/bin/dagster-components
4+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
5+
┃ Component Type ┃ Summary ┃
6+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
7+
│ definitions@dagster_components │ Wraps an arbitrary set of │
8+
│ │ Dagster definitions. │
9+
│ pipes_subprocess_script_collection@dagster_components │ Assets that wrap Python │
10+
│ │ scripts executed with │
11+
│ │ Dagster's │
12+
│ │ PipesSubprocessClient. │
13+
│ shell_command@my_component_library │ Models a shell script as a │
14+
│ │ Dagster asset. │
15+
└───────────────────────────────────────────────────────┴────────────────────────────────┘
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
dg component-type docs shell_command@my_component_library
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
2+
<!DOCTYPE html>
3+
<html lang="en">
4+
<head>
5+
<meta charset="UTF-8">
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
7+
<title>Markdown Preview</title>
8+
</head>
9+
<body>
10+
<h2>Component: <code>my_component_library.shell_command</code></h2>
11+
<h3>Description:</h3>
12+
<p>Models a shell script as a Dagster asset.</p>
13+
<h3>Sample Component Params:</h3>
14+
<textarea rows=21 cols=100>
15+
type: my_component_library.shell_command
16+
17+
params:
18+
script_path: '...'
19+
asset_specs:
20+
- deps:
21+
- '...'
22+
description: '...'
23+
metadata: '...'
24+
group_name: '...'
25+
skippable: false
26+
code_version: '...'
27+
owners:
28+
- '...'
29+
tags: '...'
30+
kinds:
31+
- '...'
32+
automation_condition: '...'
33+
key: '...'
34+
35+
</textarea>
36+
</body>
37+
</html>
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
type: shell_command@my_component_library
2+
3+
params: {}

examples/docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/with-build-defs.py

+7-3
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ def resolve_asset_specs(
3131
@registered_component_type(name="shell_command")
3232
@dataclass
3333
class ShellCommand(Component):
34+
"""Models a shell script as a Dagster asset."""
35+
3436
script_path: str
3537
asset_specs: Annotated[Sequence[dg.AssetSpec], FieldResolver(resolve_asset_specs)]
3638

@@ -39,11 +41,13 @@ def get_schema(cls) -> type[ShellScriptSchema]:
3941
return ShellScriptSchema
4042

4143
def build_defs(self, load_context: ComponentLoadContext) -> dg.Definitions:
44+
resolved_script_path = Path(load_context.path, self.script_path).absolute()
45+
4246
@dg.multi_asset(name=Path(self.script_path).stem, specs=self.asset_specs)
4347
def _asset(context: dg.AssetExecutionContext):
44-
self.execute(context)
48+
self.execute(resolved_script_path, context)
4549

4650
return dg.Definitions(assets=[_asset])
4751

48-
def execute(self, context: dg.AssetExecutionContext):
49-
subprocess.run(["sh", self.script_path], check=True)
52+
def execute(self, resolved_script_path: Path, context: dg.AssetExecutionContext):
53+
subprocess.run(["sh", str(resolved_script_path)], check=True)

examples/docs_beta_snippets/docs_beta_snippets/guides/components/shell-script-component/with-config-schema.py

+2
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ class ShellScriptSchema(ResolvableSchema):
1818

1919
@registered_component_type(name="shell_command")
2020
class ShellCommand(Component):
21+
"""Models a shell script as a Dagster asset."""
22+
2123
@classmethod
2224
def get_schema(cls) -> type[ShellScriptSchema]:
2325
return ShellScriptSchema

examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/conftest.py

+22
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
from collections.abc import Iterator
2+
13
import pytest
24

35

@@ -11,3 +13,23 @@ def pytest_addoption(parser: pytest.Parser) -> None:
1113
@pytest.fixture
1214
def update_snippets(request: pytest.FixtureRequest) -> bool:
1315
return bool(request.config.getoption("--update-snippets"))
16+
17+
18+
@pytest.fixture(scope="session")
19+
def get_selenium_driver():
20+
from selenium import webdriver
21+
22+
driver = None
23+
24+
try:
25+
26+
def _get_driver():
27+
nonlocal driver
28+
if driver is None:
29+
driver = webdriver.Chrome()
30+
return driver
31+
32+
yield _get_driver
33+
finally:
34+
if driver:
35+
driver.quit()

0 commit comments

Comments
 (0)