Skip to content

Commit ba557d7

Browse files
neverettneverett
committed
Merge branch 'master' into nikki/docs/combine-integrations-docs
2 parents 187ff2a + 099a6a4 commit ba557d7

188 files changed

Lines changed: 1924 additions & 1214 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.buildkite/buildkite-shared/lints.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,6 @@ def test_tox_passenv(tox_config):
3131
"BUILDKITE*",
3232
"PYTEST_ADDOPTS",
3333
"PYTEST_PLUGINS",
34-
"DAGSTER_GIT_REPO_DIR",
3534
]
3635

3736
missing_env = []

‎.buildkite/hooks/pre-command‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,8 @@
11
set -euo pipefail
22

3-
export DAGSTER_GIT_REPO_DIR=/workdir
4-
53
if [ -z "${PYTEST_CURRENT_TEST:-}" ]; then
64
. ./.buildkite/buildkite-shared/buildkite_shared/scripts/docker_prune.sh
75
fi
86

97
export COMMAND_START_TIME="$(date +%s)"
108
export STEP_SPAN_ID="$(echo $BUILDKITE_LABEL | sum | cut -f 1 -d \ )"
11-
12-

‎.gitignore‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -138,11 +138,6 @@ docs/gatsby/**/public
138138
# Next stuff
139139
docs/next/.mdx-data
140140
docs/next/public/sitemap.xml
141-
# Data
142-
data
143-
# Don't ignore data folders in examples
144-
!examples/*/data
145-
!examples/**/**/data
146141

147142
# Dask
148143
dask-worker-space

‎CHANGES.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,32 @@
11
# Changelog
22

3+
## 1.12.7 (core) / 0.28.7 (libraries)
4+
5+
### New
6+
7+
- Optimized performance of calculating partition keys for time window partitions with exclusions.
8+
- `timedelta` and `datetime` are now available via the `datetime` context when rendering components (Thanks, [@stevenayers](https://github.com/stevenayers)!)
9+
- `FreshnessPolicy` is now available via the `dg` context when rendering components. (Thanks, [@stevenayers](@https://github.com/stevenayers)!)
10+
- Assets may now be annotated with up to 10 kinds (limit was previously 3).
11+
- Arbitrary resource parameters may now be hidden in the UI by setting `json_schema_extra={"dagster__is_secret": True}` on the corresponding `Field` definition in the resource class.
12+
- The `dg docs` cli group has been removed. The `integrations` subcommand has been moved to `dg utils integrations`.
13+
- Bumped the `gql` dependency in `dagster-graphql` to be inclusive of v4 for broader transitive dependency compatibility
14+
- [dagster-omni] Fix issue where retries would terminate while asynchronously gathering metadata.
15+
- [dagster-tableau] The value of resource parameter `TableauWorkspace.connected_app_secret_value` is now hidden in the UI.
16+
- [dagster-tableau] Updated extraction logic to handle hidden sheets. The materializable data sources connected to these sheets are now successfully detected and included as materializable data assets. (Thanks, [@miriamcastel](https://github.com/miriamcastel)!)
17+
18+
### Bugfixes
19+
20+
- Fix an AttributeError when calling `map_asset_specs` on assets defined using the `ins` parameter. (Thanks, [@Jongwan93](https://github.com/Jongwan93)!)
21+
- Fix an issue where backfill runs that incorrectly created unpartitioned materializations of a partitioned asset, or partitioned materializations of an unpartitioned asset due to incorrect asset business logic would move the backfill into an invalid state. Now, the backfill will detect this case and fail the backfill.
22+
- Fixed an issue with `dg plus deploy refresh-defs-state` which could cause errors when refreshing state for components that required CLIs that were only available in the project environment.
23+
- [dagster-dbt] Fixed issue that could cause errors when emitting events for a dbt Cloud job run.
24+
25+
### Dagster Plus
26+
27+
- Dagster Plus Pro users can now create service users. Service users are accounts that can
28+
authenticate API requests but that are not tied to any particular human user.
29+
330
## 1.12.6 (core) / 0.28.6 (libraries)
431

532
### New

‎docs/docs/deployment/dagster-plus/authentication-and-access-control/rbac/users.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ In this guide, you'll learn how to manage users and their permissions using the
2525
- **If using Google for Single sign-on (SSO)**, users must be added in Dagster+ before they can log in.
2626
- **If using an Identity Provider (IdP) like Okta for SSO**, users must be assigned to the Dagster app in the IdP to be able to log in to Dagster+. Refer to the [SSO setup guides](/deployment/dagster-plus/authentication-and-access-control/sso) for setup instructions for each of our supported IdP solutions.
2727

28+
:::note
29+
30+
SCIM provisioning does not affect the ability to manage [service
31+
users](#service-users). Service users are always created and managed through the Dagster UI.
32+
33+
:::
34+
2835
By default, users will be granted Viewer permissions on each deployment. The default role can be adjusted by modifying the [`sso_default_role` deployment setting](/deployment/dagster-plus/deploying-code/full-deployments/full-deployment-settings-reference).
2936

3037
## Adding users to Dagster+
@@ -89,6 +96,39 @@ Removing a user removes them from the organization. **Note**: If using a SAML-ba
8996
6. Click **Remove user**.
9097
7. When prompted, confirm the removal.
9198

99+
## Service users
100+
101+
:::note
102+
103+
Service users are a Dagster+ Pro feature.
104+
105+
:::
106+
107+
Service users are non-human users that can be used to authenticate API
108+
requests, but cannot log in to the UI or be members of a team. Service users do
109+
not apply to an organization's seat cap. You can create as many as you need.
110+
111+
Service users are added, removed, and edited with the same UI as for
112+
regular users. If your organization has access to service users, the "Add New
113+
User" button above the Users table will instead be a dropdown allowing you to select
114+
between adding a new human user or a new service user.
115+
116+
![Screenshot of "Add users" dropdown](/images/dagster-plus/features/authentication-and-access-control/add-users-dropdown.png)
117+
118+
The primary identifier for a service user is its `name`, which must be unique
119+
within the organization. A description may also optionally be provided:
120+
121+
![Screenshot of "Add service user" form](/images/dagster-plus/features/authentication-and-access-control/add-service-user.png)
122+
123+
Once the service user has been created, you will be presented with the same
124+
permissions management UI as for human users:
125+
126+
![Screenshot of assigning roles to a service user](/images/dagster-plus/features/authentication-and-access-control/service-user-roles.png)
127+
128+
You will typically want to immediately create a
129+
[token](/deployment/dagster-plus/management/tokens/user-tokens) for a service user
130+
after creating it, since service users cannot do anything without a token.
131+
92132
## Next steps
93133

94134
- Learn more about role-based access control (RBAC) in [Understanding User Roles & Permissions](/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions)

‎docs/docs/deployment/dagster-plus/deploying-code/adding-project-to-dagster-plus.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: Adding a project to Dagster+
3+
sidebar_position: 30
4+
tags: [dagster-plus-feature]
5+
description: You can add a single Dagster project manually to your Dagster+ deployment or add multiple Dagster projects that roll up into one global lineage graph.
6+
---
7+
8+
import DagsterPlus from '@site/docs/partials/\_DagsterPlus.md';
9+
10+
<DagsterPlus />
11+
12+
This guide will cover three options for adding a new Dagster project (also known as a code location) to your Dagster+ deployment:
13+
14+
- [Adding a Dagster project manually](#adding-a-new-dagster-project-manually)
15+
- [Adding a Dagster project in a new Git repository](#adding-a-dagster-project-in-a-new-git-repository)
16+
- [Adding a new Dagster project to an existing Git monorepo](#adding-a-new-dagster-project-to-a-git-monorepo)
17+
18+
## Prerequisites
19+
20+
Before following the steps in this guide, you will need to:
21+
22+
- [Create a Dagster project](/guides/build/projects/creating-projects).
23+
- Have [Editor, Admin, or Organization Admin permissions in Dagster+](/deployment/dagster-plus/authentication-and-access-control/rbac/user-roles-permissions).
24+
- Find your Dagster+ deployment type. An administrator can help find this information, or you can locate it by clicking _Deployment > Agents tab_ in the Dagster UI. Dagster+ Serverless organizations will have a _Managed by Dagster+_ label next to the agent.
25+
26+
Adding a project follows two steps:
27+
28+
- For Dagster+ Hybrid, ensuring the Dagster code is in a place accessible by your agent, usually by building a Docker image with your code that's pushed to a registry. For Dagster+ Serverless you can skip this step.
29+
- Notifying Dagster+ of the new or updated code location. This will be done by using the Dagster+ Python client.
30+
31+
Often these two steps are handled by CI/CD connected to your Git repository.
32+
33+
## Adding a new Dagster project manually
34+
35+
### Step 1: Install the `dagster-cloud` Python client
36+
37+
Start by installing the `dagster-cloud` Python client:
38+
39+
<Tabs groupId="package-manager">
40+
<TabItem value="uv" label="uv">
41+
42+
```shell
43+
uv add dagster-cloud
44+
```
45+
46+
</TabItem>
47+
48+
<TabItem value="pip" label="pip">
49+
50+
```shell
51+
pip install dagster-cloud
52+
```
53+
54+
</TabItem>
55+
</Tabs>
56+
57+
### Step 2: Authenticate the `dagster-cloud` Python client
58+
59+
Next, authenticate the `dagster-cloud` Python client:
60+
61+
1. In the Dagster+ UI, click the user icon in the upper right corner.
62+
2. Click **Organization settings**, then the **Tokens** tab.
63+
3. Click the **Create user token** button.
64+
4. Copy the token.
65+
5. Set the following environment variables:
66+
67+
```bash
68+
export DAGSTER_CLOUD_ORGANIZATION="organization-name" # if your URL is https://acme.dagster.plus your organization name is "acme"
69+
export DAGSTER_CLOUD_API_TOKEN="your-token"
70+
```
71+
72+
### Step 3: Add your Dagster project
73+
74+
Now add your Dagster project. The following example assumes you are running the command from the top-level working directory of your Dagster project with a project named "dagster_tutorial" structured as a Python module named "dagster_tutorial":
75+
76+
<Tabs groupId="package-manager">
77+
78+
<TabItem value="uv" label="uv">
79+
```shell
80+
.
81+
├── pyproject.toml
82+
├── README.md
83+
├── src
84+
│   └── dagster_tutorial
85+
│   ├── __init__.py
86+
│   ├── definitions.py
87+
│   └── defs
88+
│   └── __init__.py
89+
├── tests
90+
│   └── __init__.py
91+
└── uv.lock
92+
```
93+
</TabItem>
94+
<TabItem value="pip" label="pip">
95+
```shell
96+
.
97+
├── pyproject.toml
98+
├── README.md
99+
├── src
100+
│   └── dagster_tutorial
101+
│   ├── __init__.py
102+
│   ├── definitions.py
103+
│   └── defs
104+
│   └── __init__.py
105+
└── tests
106+
   └── __init__.py
107+
```
108+
</TabItem>
109+
</Tabs>
110+
111+
The commands below take two main arguments:
112+
113+
- `module_name` is determined by your code structure
114+
- `location_name` is the unique label for this project used in Dagster+
115+
116+
<Tabs>
117+
<TabItem value="serverless" label="Dagster+ Serverless">
118+
119+
If you are using Dagster+ Serverless, run the following command to add your Dagster project:
120+
121+
```bash
122+
dagster-cloud serverless deploy-python-executable --deployment prod --location-name dagster_tutorial --module-name dagster_tutorial
123+
```
124+
125+
Running the command multiple times with the same location name will _update_ the project. Running the command with a new location name will _add_ a project.
126+
127+
</TabItem>
128+
<TabItem value="hybrid" label="Dagster+ Hybrid">
129+
130+
If you are using Dagster+ Hybrid, make sure you have deployed the code appropriately by:
131+
132+
1. Building a Docker image and pushing it to an image registry.
133+
2. Running the `dg plus deploy configure` command from the Dagster project root to create a `dagster_cloud.yaml` file.
134+
135+
Then run the following command, using the image URI which is available from your registry:
136+
137+
```bash
138+
dagster-cloud deployment add-location --deployment prod --location-name dagster_tutorial --module-name dagster_tutorial --image 764506304434.dkr.ecr.us-west-2.amazonaws.com/hooli-data-science-prod:latest
139+
```
140+
141+
</TabItem>
142+
</Tabs>
143+
144+
After running the command, you can verify the project was deployed by navigating to the _Deployments_ tab on Dagster+.
145+
146+
## Adding a Dagster project in a new Git repository
147+
148+
Adding a project to a Git repository follows the same steps as adding a project manually, but automates those steps by running them through CI/CD. For more information, see [Configuring CI/CD in Dagster+](/deployment/dagster-plus/deploying-code/configuring-ci-cd).
149+
150+
The CI/CD process will add the project to Dagster+, which can be verified by viewing the _Deployments_ tab.
151+
152+
## Adding a new Dagster project to a Git monorepo
153+
154+
Many organizations use a Git monorepo to contain multiple Dagster projects. To add a new Dagster project to a monorepo:
155+
156+
1. [Create a Dagster workspace](/guides/build/projects/workspaces/creating-workspaces) to hold your Dagster projects.
157+
2. Move your existing Dagster project(s) into the `/projects` directory of the workspace. If necessary, [convert the projects](/guides/build/projects/moving-to-components/migrating-project) to the new Dagster project structure.
158+
3. In the `/projects` directory of the workspace, [create a new Dagster project](/guides/build/projects/creating-projects) with the `dagster-create` CLI.
159+
4. Run the `dg plus deploy configure` command in the workspace root to create deployment configuration files for projects that need them.
160+
161+
After following these steps, trigger the CI/CD process to add your project to Dagster+. Navigate to the _Deployments_ tab in Dagster+ to confirm your project was added.
162+
163+
## Next steps
164+
165+
- After adding a code location, you may want to set up [access controls](/deployment/dagster-plus/authentication-and-access-control)
166+
- You may want to add additional configuration to your project. This configuration will vary by agent type, but see examples for [setting default resource limits for Kubernetes](/deployment/dagster-plus/hybrid/kubernetes) or [changing the IAM role for ECS](/deployment/dagster-plus/hybrid/amazon-ecs/configuration-reference).

‎docs/docs/deployment/dagster-plus/deploying-code/branch-deployments/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Branch deployments
3-
sidebar_position: 30
3+
sidebar_position: 40
44
description: With branch deployments, Dagster+ creates a corresponding ephemeral preview deployment of your Dagster code for each pull or merge request to show what your pipeline will look like after the change is merged.
55
canonicalUrl: '/deployment/dagster-plus/deploying-code/branch-deployments'
66
slug: '/deployment/dagster-plus/deploying-code/branch-deployments'

‎docs/docs/deployment/dagster-plus/deploying-code/configuring-ci-cd.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: Configuring CI/CD in Dagster+
33
sidebar_label: CI/CD
44
description: Implement CI/CD for your Dagster+ Serverless or Hybrid deployment with GitHub, GitLab, or another CI/CD provider.
5-
sidebar_position: 20
5+
sidebar_position: 30
66
tags: [dagster-plus-feature]
77
---
88

‎docs/docs/deployment/dagster-plus/management/tokens/user-tokens.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ tags: [dagster-plus-feature]
77

88
import ThemedImage from '@theme/ThemedImage';
99

10-
In this guide, we'll walk you through creating, viewing, editing, and revoking user tokens in Dagster+.
10+
In this guide, we'll walk you through creating, viewing, editing, and revoking user tokens in Dagster+. The below guide applies to both human users and [service users](/deployment/dagster-plus/authentication-and-access-control/rbac/users#service-users).
1111

1212
## Creating and revoking your own user tokens
1313

‎docs/docs/guides/build/partitions-and-backfills/backfilling-data.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@ Backfills can also be launched for a selection of partitioned assets as long as
1818

1919
![Backfills launch modal](/images/guides/build/partitions-and-backfills/asset-backfill-partition-selection-modal.png)
2020

21+
:::note
22+
23+
The "Launch as..." toggle shown in the screenshot above (for choosing between single run and multiple runs) was removed from the UI in Dagster 1.5. Single-run backfills are now configured in code using `backfill_policy`. See [Launching single-run backfills using backfill policies](#launching-single-run-backfills-using-backfill-policies) below.
24+
25+
:::
26+
2127
To observe the progress of an asset backfill, navigate to the **Runs details** page for the run. This page can be accessed by clicking **Runs tab**, then clicking the ID of the run. To see all runs, including runs launched by a backfill, check the **Show runs within backfills** box:
2228

2329
![Asset backfill details page](/images/guides/build/partitions-and-backfills/asset-backfill-details-page.png)

0 commit comments

Comments
 (0)