Skip to content

Commit 00d64dd

Browse files
petehuntcmpaddenalangenfeldjmsanders
authored
RFC: Add evidence.dev to the components tutorial (#29074)
## Summary & Motivation I think showing a nice data visualization would be a great payoff for the components tutorial. I implemented a Component for evidence.dev here: dagster-io/community-integrations#167, which needs to land before this PR can be updated and landed. We should also move the dashboard example itself to the dagster org if this is accepted https://github.com/petehunt/jaffle_dashboard ## How I Tested These Changes ## Changelog > Insert changelog entry or delete this section. --------- Co-authored-by: Colton Padden <[email protected]> Co-authored-by: Alex Langenfeld <[email protected]> Co-authored-by: Jordan Sanders <[email protected]>
1 parent 5f49687 commit 00d64dd

25 files changed

+352
-14
lines changed

‎docs/docs/guides/labs/components/components-etl-pipeline-tutorial.md

+54-2
Original file line numberDiff line numberDiff line change
@@ -220,18 +220,70 @@ To verify the fix, you can view a sample of the newly materialized assets in Duc
220220

221221
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/24-duckdb-select-orders.txt" />
222222

223+
## Visualize data
224+
225+
To visualize the data we've just transformed we'll use [Evidence.dev](https://www.evidence.dev/), an open-source BI tool.
226+
227+
### 1. Install the `dagster-evidence` package
228+
229+
<CliInvocationExample contents="uv add dagster-evidence" />
230+
231+
You will see that the `dagster-evidence` package provides a new `EvidenceProject` component type:
232+
233+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/25-dg-list-component-types.txt" />
234+
235+
### 2. Clone a sample Evidence project from GitHub
236+
237+
Clone the example dashboard project, and be sure to install the dependencies with `cd jaffle_dashboard && npm install`.
238+
239+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/26-jaffle-dashboard-clone.txt" />
240+
241+
### 3. Scaffold a new instance of the Evidence project component
242+
243+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/27-scaffold-jaffle-dashboard.txt" />
244+
245+
It will generate an empty YAML file:
246+
247+
<CodeExample
248+
path="docs_snippets/docs_snippets/guides/components/index/28-component-jaffle-dashboard.yaml"
249+
language="YAML"
250+
title="jaffle-platform/jaffle_platform/defs/jaffle_dashboard/component.yaml"
251+
/>
252+
253+
### 4. Configure the component
254+
255+
Let's update the configuration of the component to target the `jaffle_dashboard` Evidence project, and wire it up to our two upstream assets:
256+
257+
<CodeExample
258+
path="docs_snippets/docs_snippets/guides/components/index/29-project-jaffle-dashboard.yaml"
259+
language="YAML"
260+
title="jaffle-platform/jaffle_platform/defs/jaffle_dashboard/component.yaml"
261+
/>
262+
263+
And let's verify that the YAML is correct:
264+
265+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/30-dg-component-check-yaml.txt" />
266+
267+
And that the definitions load successfully:
268+
269+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/31-dg-component-check-defs.txt" />
270+
271+
Materialize the Evidence assets in the UI, and it will generate a static website for your dashboard in the `build` directory. You can view the dashboard in your browser by running `python -m http.server` in that directory, which should result in something that looks like this:
272+
273+
![](/images/guides/build/projects-and-components/components/evidence.png)
274+
223275
## Automate the pipeline
224276

225277
Now that you've defined some assets, let's schedule them.
226278

227279
First scaffold in a schedule:
228280

229-
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/25-scaffold-daily-jaffle.txt" />
281+
<CliInvocationExample path="docs_snippets/docs_snippets/guides/components/index/32-scaffold-daily-jaffle.txt" />
230282

231283
And now target `*` and schedule `@daily`:
232284

233285
<CodeExample
234-
path="docs_snippets/docs_snippets/guides/components/index/26-daily-jaffle.py"
286+
path="docs_snippets/docs_snippets/guides/components/index/33-daily-jaffle.py"
235287
language="Python"
236288
title="jaffle-platform/src/jaffle_platform/defs/daily_jaffle.py"
237289
/>

‎docs/static/images/guides/build/projects-and-components/components/evidence.png

101 KB
Loading

‎examples/docs_snippets/docs_snippets/guides/components/index/25-dg-list-component-types.txt

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
dg list component-type
2+
3+
4+
warnings.warn(message)
5+
Using /.../jaffle-platform/.venv/bin/dagster-components
6+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
7+
┃ Component Type ┃ Summary ┃
8+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
9+
│ dagster.components.DefinitionsComponent │ An arbitrary set of │
10+
│ │ dagster definitions. │
11+
│ dagster.components.DefsFolderComponent │ A folder which may │
12+
│ │ contain multiple │
13+
│ │ submodules, each │
14+
│ │ which define components. │
15+
│ dagster.components.PipesSubprocessScriptCollectionComponent │ Assets that wrap Python │
16+
│ │ scripts executed with │
17+
│ │ Dagster's │
18+
│ │ PipesSubprocessClient. │
19+
│ dagster_dbt.DbtProjectComponent │ Expose a DBT project to │
20+
│ │ Dagster as a set of │
21+
│ │ assets. │
22+
│ dagster_evidence.EvidenceProject │ Expose an Evidence.dev │
23+
│ │ dashboard as a Dagster │
24+
│ │ asset. │
25+
│ dagster_sling.SlingReplicationCollectionComponent │ Expose one or more Sling │
26+
│ │ replications to Dagster │
27+
│ │ as assets. │
28+
└─────────────────────────────────────────────────────────────┴──────────────────────────┘

‎examples/docs_snippets/docs_snippets/guides/components/index/25-dg-list-plugins.txt

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
dg list plugins
2+
3+
4+
warnings.warn(message)
5+
Using /.../jaffle-platform/.venv/bin/dagster-components
6+
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
7+
┃ Plugin ┃ Objects ┃
8+
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
9+
│ dagster │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ │
10+
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
11+
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │
12+
│ │ │ dagster.asset │ Create a │ [scaffold-tar… │ │
13+
│ │ │ │ definition for │ │ │
14+
│ │ │ │ how to compute │ │ │
15+
│ │ │ │ an asset. │ │ │
16+
│ │ ├─────────────────────────────────────────────────────────────┼────────────────┼────────────────┤ │
17+
│ │ │ dagster.components.DefinitionsComponent │ An arbitrary │ [component, │ │
18+
│ │ │ │ set of dagster │ scaffold-targ… │ │
19+
│ │ │ │ definitions. │ │ │
20+
│ │ ├─────────────────────────────────────────────────────────────┼────────────────┼────────────────┤ │
21+
│ │ │ dagster.components.DefsFolderComponent │ A folder which │ [component, │ │
22+
│ │ │ │ may contain │ scaffold-targ… │ │
23+
│ │ │ │ multiple │ │ │
24+
│ │ │ │ submodules, │ │ │
25+
│ │ │ │ each │ │ │
26+
│ │ │ │ which define │ │ │
27+
│ │ │ │ components. │ │ │
28+
│ │ ├─────────────────────────────────────────────────────────────┼────────────────┼────────────────┤ │
29+
│ │ │ dagster.components.PipesSubprocessScriptCollectionComponent │ Assets that │ [component, │ │
30+
│ │ │ │ wrap Python │ scaffold-targ… │ │
31+
│ │ │ │ scripts │ │ │
32+
│ │ │ │ executed with │ │ │
33+
│ │ │ │ Dagster's │ │ │
34+
│ │ │ │ PipesSubproce… │ │ │
35+
│ │ ├─────────────────────────────────────────────────────────────┼────────────────┼────────────────┤ │
36+
│ │ │ dagster.schedule │ Creates a │ [scaffold-tar… │ │
37+
│ │ │ │ schedule │ │ │
38+
│ │ │ │ following the │ │ │
39+
│ │ │ │ provided cron │ │ │
40+
│ │ │ │ schedule and │ │ │
41+
│ │ │ │ requests runs │ │ │
42+
│ │ │ │ for the │ │ │
43+
│ │ │ │ provided job. │ │ │
44+
│ │ ├─────────────────────────────────────────────────────────────┼────────────────┼────────────────┤ │
45+
│ │ │ dagster.sensor │ Creates a │ [scaffold-tar… │ │
46+
│ │ │ │ sensor where │ │ │
47+
│ │ │ │ the decorated │ │ │
48+
│ │ │ │ function is │ │ │
49+
│ │ │ │ used as the │ │ │
50+
│ │ │ │ sensor's │ │ │
51+
│ │ │ │ evaluation │ │ │
52+
│ │ │ │ function. │ │ │
53+
│ │ └─────────────────────────────────────────────────────────────┴────────────────┴────────────────┘ │
54+
│ dagster_dbt │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
55+
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
56+
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
57+
│ │ │ dagster_dbt.DbtProjectComponent │ Expose a DBT project to │ [component, scaffold-target] │ │
58+
│ │ │ │ Dagster as a set of assets. │ │ │
59+
│ │ └─────────────────────────────────┴──────────────────────────────┴──────────────────────────────┘ │
60+
│ dagster_evidence │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
61+
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
62+
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
63+
│ │ │ dagster_evidence.EvidenceProject │ Expose an Evidence.dev │ [component, │ │
64+
│ │ │ │ dashboard as a Dagster │ scaffold-target] │ │
65+
│ │ │ │ asset. │ │ │
66+
│ │ └──────────────────────────────────┴──────────────────────────────┴─────────────────────────────┘ │
67+
│ dagster_sling │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓ │
68+
│ │ ┃ Symbol ┃ Summary ┃ Features ┃ │
69+
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩ │
70+
│ │ │ dagster_sling.SlingReplicationCollectionComponent │ Expose one or more │ [component, │ │
71+
│ │ │ │ Sling replications │ scaffold-target] │ │
72+
│ │ │ │ to Dagster as │ │ │
73+
│ │ │ │ assets. │ │ │
74+
│ │ └───────────────────────────────────────────────────┴─────────────────────┴─────────────────────┘ │
75+
└──────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

‎examples/docs_snippets/docs_snippets/guides/components/index/26-jaffle-dashboard-clone.txt

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
git clone --depth=1 https://github.com/petehunt/jaffle_dashboard.git jaffle_dashboard && rm -rf jaffle_dashboard/.git

0 commit comments

Comments
 (0)