Skip to content

Commit 46a9fdb

Browse files
neverettbraunjj
neverett
authored and
braunjj
committed
DOC-591 Switch embedded example code to CodeExample component (#27605)
## Summary & Motivation See title -- need to do this before re-implementing versioned docs. ## How I Tested These Changes ## Changelog > Insert changelog entry or delete this section. --------- Signed-off-by: nikki everett <[email protected]>
1 parent 74bc3b5 commit 46a9fdb

File tree

67 files changed

+384
-7175
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+384
-7175
lines changed

Diff for: ‎docs/docs-beta/CONTRIBUTING.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ To include code snippets, use the following format:
129129
<CodeExample path="path/to/file.py" />
130130
```
131131

132-
You can optionally include [additional properties](https://github.com/dagster-io/dagster/blob/master/docs/docs-beta/src/components/CodeExample.tsx#L4), such as `language`, `title`, `lineStart`, `lineEnd`, `startAfter`, and `endBefore`:
132+
You can optionally include [additional properties](https://github.com/dagster-io/dagster/blob/master/docs/docs-beta/src/components/CodeExample.tsx#L6), such as `language`, `title`, `lineStart`, `lineEnd`, `startAfter`, and `endBefore`:
133133

134134
```
135135
<CodeExample path="path/to/file.py" language="python" startAfter="start-after-comment" endBefore="end-before-comment" title="My example" />

Diff for: ‎docs/docs-beta/docs/dagster-plus/features/ci-cd/branch-deployments/testing.md

+9-320
Original file line numberDiff line numberDiff line change
@@ -58,54 +58,7 @@ To set up a branch deployment workflow to construct and test these tables, we wi
5858

5959
In production, we want to write three tables to Snowflake: `ITEMS`, `COMMENTS`, and `STORIES`. We can define these tables as assets as follows:
6060

61-
{/* TODO convert to <CodeExample> */}
62-
```python file=/guides/dagster/development_to_production/assets.py startafter=start_assets endbefore=end_assets
63-
# assets.py
64-
import pandas as pd
65-
import requests
66-
67-
from dagster import Config, asset
68-
69-
70-
class ItemsConfig(Config):
71-
base_item_id: int
72-
73-
74-
@asset(
75-
io_manager_key="snowflake_io_manager",
76-
)
77-
def items(config: ItemsConfig) -> pd.DataFrame:
78-
"""Items from the Hacker News API: each is a story or a comment on a story."""
79-
rows = []
80-
max_id = requests.get(
81-
"https://hacker-news.firebaseio.com/v0/maxitem.json", timeout=5
82-
).json()
83-
# Hacker News API is 1-indexed, so adjust range by 1
84-
for item_id in range(max_id - config.base_item_id + 1, max_id + 1):
85-
item_url = f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
86-
rows.append(requests.get(item_url, timeout=5).json())
87-
88-
# ITEM_FIELD_NAMES is a list of the column names in the Hacker News dataset
89-
result = pd.DataFrame(rows, columns=ITEM_FIELD_NAMES).drop_duplicates(subset=["id"])
90-
result.rename(columns={"by": "user_id"}, inplace=True)
91-
return result
92-
93-
94-
@asset(
95-
io_manager_key="snowflake_io_manager",
96-
)
97-
def comments(items: pd.DataFrame) -> pd.DataFrame:
98-
"""Comments from the Hacker News API."""
99-
return items[items["type"] == "comment"]
100-
101-
102-
@asset(
103-
io_manager_key="snowflake_io_manager",
104-
)
105-
def stories(items: pd.DataFrame) -> pd.DataFrame:
106-
"""Stories from the Hacker News API."""
107-
return items[items["type"] == "story"]
108-
```
61+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/assets.py" startAfter="start_assets" endBefore="end_assets" />
10962

11063
As you can see, our assets use an [I/O manager](/guides/build/io-managers/) named `snowflake_io_manager`. Using I/O managers and other resources allow us to swap out implementations per environment without modifying our business logic.
11164

@@ -119,46 +72,7 @@ Dagster automatically sets certain [environment variables](/dagster-plus/deploym
11972

12073
Because we want to configure our assets to write to Snowflake using a different set of credentials and database in each environment, we'll configure a separate I/O manager for each environment:
12174

122-
{/* TODO convert to <CodeExample> */}
123-
```python file=/guides/dagster/development_to_production/branch_deployments/repository_v1.py startafter=start_repository endbefore=end_repository
124-
# definitions.py
125-
from dagster import Definitions
126-
127-
from ..assets import comments, items, stories
128-
129-
snowflake_config = {
130-
"account": "abc1234.us-east-1",
131-
"user": "[email protected]",
132-
"password": {"env": "SYSTEM_SNOWFLAKE_PASSWORD"},
133-
"schema": "HACKER_NEWS",
134-
}
135-
136-
resources = {
137-
"branch": {
138-
"snowflake_io_manager": SnowflakePandasIOManager(
139-
**snowflake_config,
140-
database=f"PRODUCTION_CLONE_{os.getenv('DAGSTER_CLOUD_PULL_REQUEST_ID')}",
141-
),
142-
},
143-
"prod": {
144-
"snowflake_io_manager": SnowflakePandasIOManager(
145-
**snowflake_config,
146-
database="PRODUCTION",
147-
),
148-
},
149-
}
150-
151-
152-
def get_current_env():
153-
is_branch_depl = os.getenv("DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT") == "1"
154-
assert is_branch_depl is not None # env var must be set
155-
return "branch" if is_branch_depl else "prod"
156-
157-
158-
defs = Definitions(
159-
assets=[items, comments, stories], resources=resources[get_current_env()]
160-
)
161-
```
75+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/repository_v1.py" startAfter="start_repository" endBefore="end_repository" />
16276

16377
Refer to the [Dagster+ environment variables documentation](/dagster-plus/deployment/management/environment-variables/) for more info about available environment variables.
16478

@@ -177,89 +91,17 @@ these tasks, like viewing them in the Global Asset Graph.
17791

17892
:::
17993

180-
{/* TODO convert to <CodeExample> */}
181-
```python file=/guides/dagster/development_to_production/branch_deployments/clone_and_drop_db.py startafter=start_clone_db endbefore=end_clone_db
182-
from dagster_snowflake import SnowflakeResource
183-
184-
from dagster import In, Nothing, graph, op
185-
186-
187-
@op
188-
def drop_database_clone(snowflake: SnowflakeResource):
189-
with snowflake.get_connection() as conn:
190-
cur = conn.cursor()
191-
cur.execute(
192-
"DROP DATABASE IF EXISTS"
193-
f" PRODUCTION_CLONE_{os.environ['DAGSTER_CLOUD_PULL_REQUEST_ID']}"
194-
)
195-
196-
197-
@op(ins={"start": In(Nothing)})
198-
def clone_production_database(snowflake: SnowflakeResource):
199-
with snowflake.get_connection() as conn:
200-
cur = conn.cursor()
201-
cur.execute(
202-
"CREATE DATABASE"
203-
f" PRODUCTION_CLONE_{os.environ['DAGSTER_CLOUD_PULL_REQUEST_ID']} CLONE"
204-
' "PRODUCTION"'
205-
)
206-
207-
208-
@graph
209-
def clone_prod():
210-
clone_production_database(start=drop_database_clone())
211-
212-
213-
@graph
214-
def drop_prod_clone():
215-
drop_database_clone()
216-
```
94+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_and_drop_db.py" startAfter="start_clone_db" endBefore="end_clone_db" />
21795

21896
We've defined `drop_database_clone` and `clone_production_database` to utilize the <PyObject section="libraries" object="SnowflakeResource" module="dagster_snowflake" />. The Snowflake resource will use the same configuration as the Snowflake I/O manager to generate a connection to Snowflake. However, while our I/O manager writes outputs to Snowflake, the Snowflake resource executes queries against Snowflake.
21997

22098
We now need to define resources that configure our jobs to the current environment. We can modify the resource mapping by environment as follows:
22199

222-
{/* TODO convert to <CodeExample> */}
223-
```python file=/guides/dagster/development_to_production/branch_deployments/repository_v2.py startafter=start_resources endbefore=end_resources
224-
resources = {
225-
"branch": {
226-
"snowflake_io_manager": SnowflakePandasIOManager(
227-
**snowflake_config,
228-
database=f"PRODUCTION_CLONE_{os.getenv('DAGSTER_CLOUD_PULL_REQUEST_ID')}",
229-
),
230-
"snowflake": SnowflakeResource(
231-
**snowflake_config,
232-
database=f"PRODUCTION_CLONE_{os.getenv('DAGSTER_CLOUD_PULL_REQUEST_ID')}",
233-
),
234-
},
235-
"prod": {
236-
"snowflake_io_manager": SnowflakePandasIOManager(
237-
**snowflake_config,
238-
database="PRODUCTION",
239-
),
240-
"snowflake": SnowflakeResource(**snowflake_config, database="PRODUCTION"),
241-
},
242-
}
243-
```
100+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/repository_v2.py" startAfter="start_resources" endBefore="end_resources" />
244101

245102
Then, we can add the `clone_prod` and `drop_prod_clone` jobs that now use the appropriate resource to the environment and add them to our definitions:
246103

247-
{/* TODO convert to <CodeExample> */}
248-
```python file=/guides/dagster/development_to_production/branch_deployments/repository_v2.py startafter=start_repository endbefore=end_repository
249-
branch_deployment_jobs = [
250-
clone_prod.to_job(),
251-
drop_prod_clone.to_job(),
252-
]
253-
defs = Definitions(
254-
assets=[items, comments, stories],
255-
resources=resources[get_current_env()],
256-
jobs=(
257-
branch_deployment_jobs
258-
if os.getenv("DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT") == "1"
259-
else []
260-
),
261-
)
262-
```
104+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/repository_v2.py" startAfter="start_repository" endBefore="end_repository" />
263105

264106
## Step 4: Create our database clone upon opening a branch
265107

@@ -268,37 +110,7 @@ defs = Definitions(
268110

269111
The `branch_deployments.yml` file located in `.github/workflows/branch_deployments.yml` defines a `dagster_cloud_build_push` job with a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end `dagster_cloud_build_push`. This job is triggered on multiple pull request events: `opened`, `synchronize`, `reopen`, and `closed`. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`. The `if` condition below ensures that `clone_prod` will not run if the pull request is closed:
270112

271-
{/* TODO convert to <CodeExample> */}
272-
```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml
273-
# .github/workflows/branch_deployments.yml
274-
275-
name: Dagster Branch Deployments
276-
on:
277-
pull_request:
278-
types: [opened, synchronize, reopened, closed]
279-
env:
280-
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
281-
282-
jobs:
283-
dagster_cloud_build_push:
284-
runs-on: ubuntu-latest
285-
name: Dagster Branch Deployments
286-
strategy:
287-
...
288-
steps:
289-
# Existing steps here
290-
...
291-
- name: Clone Snowflake schema upon launch
292-
if: github.event.action != 'closed'
293-
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
294-
with:
295-
location_name: ${{ matrix.location.name }}
296-
deployment: ${{ steps.deploy.outputs.deployment }}
297-
job_name: clone_prod
298-
env:
299-
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
300-
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
301-
```
113+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.yaml" />
302114

303115
Opening a pull request for our current branch will automatically kick off a branch deployment. After the deployment launches, we can confirm that the `clone_prod` job has run:
304116

@@ -315,53 +127,7 @@ We can also view our database in Snowflake to confirm that a clone exists for ea
315127

316128
The `.gitlab-ci.yaml` script contains a `deploy` job that defines a series of steps that launch a branch deployment. Because we want to queue a run of `clone_prod` within each deployment after it launches, we'll add an additional step at the end of `deploy`. This job is triggered on when a merge request is created or updated. This means that upon future pushes to the branch, we'll trigger a run of `clone_prod`.
317129

318-
```yaml file=/guides/dagster/development_to_production/branch_deployments/clone_prod.gitlab-ci.yml
319-
# .gitlab-ci.yml
320-
321-
stages:
322-
- setup
323-
- build
324-
- deploy
325-
326-
workflow:
327-
rules:
328-
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
329-
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
330-
331-
parse-workspace:
332-
...
333-
334-
build-image:
335-
...
336-
337-
deploy-docker:
338-
...
339-
340-
deploy-docker-branch:
341-
stage: deploy
342-
rules:
343-
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
344-
dependencies:
345-
- build-image
346-
- parse-workspace
347-
image: ghcr.io/dagster-io/dagster-cloud-action:0.1.23
348-
script:
349-
# Existing steps here
350-
...
351-
352-
# Add a step to launch the job cloning the prod db
353-
- dagster-plus job launch
354-
--url "$DAGSTER_CLOUD_URL/$DEPLOYMENT_NAME"
355-
--api-token "$DAGSTER_CLOUD_API_TOKEN"
356-
--location "location_name_containing_clone_prod_job"
357-
--job clone_prod
358-
environment:
359-
name: branch/$CI_COMMIT_REF_NAME
360-
on_stop: close_branch
361-
362-
close_branch:
363-
...
364-
```
130+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/clone_prod.gitlab-ci.yml" />
365131

366132
Opening a merge request for our current branch will automatically kick off a branch deployment. After the deployment launches, we can confirm that the `clone_prod` job has run:
367133

@@ -382,91 +148,14 @@ We can also view our database in Snowflake to confirm that a clone exists for ea
382148

383149
Finally, we can add a step to our `branch_deployments.yml` file that queues a run of our `drop_prod_clone` job:
384150

385-
{/* TODO convert to <CodeExample> */}
386-
```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml
387-
# .github/workflows/branch_deployments.yml
388-
389-
name: Dagster Branch Deployments
390-
on:
391-
pull_request:
392-
types: [opened, synchronize, reopened, closed]
393-
env:
394-
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
395-
396-
jobs:
397-
dagster_cloud_build_push:
398-
runs-on: ubuntu-latest
399-
name: Dagster Branch Deployments
400-
strategy:
401-
...
402-
steps:
403-
# Existing steps here
404-
...
405-
- name: Clone Snowflake schema upon launch
406-
...
407-
- name: Delete schema clone upon PR close
408-
if: github.event.action == 'closed'
409-
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
410-
with:
411-
location_name: ${{ matrix.location.name }}
412-
deployment: ${{ steps.deploy.outputs.deployment }}
413-
job_name: drop_prod_clone
414-
env:
415-
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
416-
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}
417-
```
151+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.yaml" />
418152

419153
</TabItem>
420154
<TabItem value="Using Gitlab CI/CD">
421155

422156
Finally, we can add a step to our `.gitlab-ci.yml` file that queues a run of our `drop_prod_clone` job:
423157

424-
{/* TODO convert to <CodeExample> */}
425-
```yaml file=/guides/dagster/development_to_production/branch_deployments/drop_db_clone.gitlab-ci.yml
426-
# .gitlab-ci.yml
427-
428-
stages:
429-
- setup
430-
- build
431-
- deploy
432-
433-
workflow:
434-
rules:
435-
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
436-
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
437-
438-
parse-workspace:
439-
...
440-
441-
build-image:
442-
...
443-
444-
deploy-docker:
445-
...
446-
447-
deploy-docker-branch:
448-
...
449-
450-
close_branch:
451-
stage: deploy
452-
image: ghcr.io/dagster-io/dagster-cloud-action:0.1.23
453-
when: manual
454-
only:
455-
- merge_requests
456-
script:
457-
# Existing steps here
458-
...
459-
460-
# Add a step to launch the job dropping the cloned db
461-
- dagster-plus job launch
462-
--url "$DAGSTER_CLOUD_URL/$DEPLOYMENT_NAME"
463-
--api-token "$DAGSTER_CLOUD_API_TOKEN"
464-
--location "location_name_containing_drop_prod_clone_job"
465-
--job drop_prod_clone
466-
environment:
467-
name: branch/$CI_COMMIT_REF_NAME
468-
action: stop
469-
```
158+
<CodeExample path="docs_snippets/docs_snippets/guides/dagster/development_to_production/branch_deployments/drop_db_clone.gitlab-ci.yml" />
470159

471160
</TabItem>
472161
</Tabs>

0 commit comments

Comments
 (0)