Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ To override a code location role for an individual user:
3. Click the toggle to the left of the deployment to open a list of code locations.
4. Next to a code location, click **Edit user role**.
5. Select the user role for the code location: {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override.png" */}
5. Select the user role for the code location.
6. Click **Save**.

#### Team members
Expand All @@ -118,13 +119,11 @@ The above also applies to code locations and Branch Deployment roles.

#### Viewing overrides

To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment:

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/user-overrides-popup.png" */}

If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides:

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override-popup.png" */}
To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment. A tooltip will display any overrides that apply at the deployment level.

If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides.

#### Removing overrides

Expand Down
3 changes: 1 addition & 2 deletions docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,8 @@ helm --namespace dagster-cloud upgrade agent \

## Troubleshooting tips

You can see basic health information about your agent in the Dagster+ UI:

[comment]: <> (TODO: Screenshot of Dagster+ Deployments agents tab)
You can see basic health information about your agent in the Dagster+ UI.

### View logs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ Now copy the following sensor code in the `sensors.py` file:
2. Navigate to the Automation page.
3. Turn on the `adhoc_request_sensor`.
4. Click on the `adhoc_request_sensor` details.

{/* TODO: Screenshot */}
You should see the sensor details view with recent ticks and the run history panel.

5. Create a `data/requests` directory in `dagster_tutorial`. Then include a `request.json` file:

Expand All @@ -76,8 +76,8 @@ Now copy the following sensor code in the `sensors.py` file:
```

6. Click on the green tick to see the run for this request.

{/* TODO: Screenshot */}
The run should appear in the runs list, and the tick details will show the run ID.

## Summary

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Then we can execute this job and pull the attributes we need to build the `conte
/>

{/* TODO the methods and statuses below do not exist in API docs */}
{/* We have provided convenience functions <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_success_event" /> and <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_failure_event" /> for retrieving `DagsterRunStatus.SUCCESS` and `DagsterRunStatus.FAILURE` events, respectively. If you have a run status sensor triggered on another status, you can retrieve all events from `result` and filter based on your event type. */}
To inspect the resulting status in tests, use the <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" /> returned by `execute_in_process`. Its `dagster_run` and `all_events` properties let you filter for the status or event type you care about.

We can use the same pattern to build the context for <PyObject section="schedules-sensors" module="dagster" object="run_failure_sensor" />. If we wanted to test this run failure sensor:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Dagster's versioning system helps you determine ahead of time whether materializ
Dagster has two versioning concepts to represent the code and input data used for each materialization:

- **Code version.** A string that represents the version of the code that computes an asset. This is the `code_version` argument of <PyObject section="assets" module="dagster" object="asset" decorator />.
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a `DataVersion` object.
{/* TODO link `DataVersion` to API docs */}
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a <PyObject section="assets" module="dagster" object="DataVersion" /> object.

By keeping track of code and data versions, Dagster can predict whether a materialization will change the underlying value. This allows Dagster to skip redundant materializations and instead return the previously computed value. In more technical terms, Dagster offers a limited form of [memoization](https://en.wikipedia.org/wiki/Memoization) for assets: the last-computed asset value is always cached.

Expand Down
4 changes: 4 additions & 0 deletions docs/docs/guides/build/assets/passing-data-between-assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,7 @@ The downsides of this approach are:
## Related resources

{/* TODO: add links to relevant API documentation here. */}
- <PyObject section="assets" module="dagster" object="AssetIn" />
- <PyObject section="assets" module="dagster" object="AssetOut" />
- <PyObject section="io-managers" module="dagster" object="IOManager" />
- <PyObject section="execution" module="dagster" object="AssetExecutionContext" />
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,18 @@ unlisted: true
---

TODO
This page lists common checks when a component fails to load or behaves unexpectedly.

## Quick checks

- **Validate the component type string**. Ensure the type name matches the registered component class and that the package providing it is installed in your environment.
- **Confirm configuration fields**. Component configuration is validated on load. If a field is missing or mis-typed, the error message will include the field path to update.
- **Verify workspace paths**. If you are using a workspace, confirm that `dg.toml` points at the correct project directories and that the project loads on its own.

## Inspecting load errors

If a component fails to load, the code location will show a load error in the UI and in CLI output. Use the stack trace there to locate the file and configuration entry that caused the failure.

## Next steps

For a refresher on how components are structured, see the [components overview](/guides/build/components).
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,8 @@ Here's what we did in this example:

This argument is passed to the `run` method of <PyObject section="libraries" integration="aws" module="dagster_aws" object="pipes.PipesLambdaClient" />, which submits the provided information to the [boto `invoke` API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda/client/invoke.html) and then invokes the specified function (`function_name`).

- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the execution in AWS Lambda has completed.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="libraries" module="pipes" object="dagster.PipesClientCompletedInvocation" /> */}
- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the execution in AWS Lambda has completed.
Comment thread
Vamsi-klu marked this conversation as resolved.

### Step 2.2: Create Dagster Definitions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,8 @@ Let's review what's happening in this code:

- **Passes the `SubmitTask` object, `AssetExecutionContext`, and `extras` dictionary to the `run` method of <PyObject section="libraries" integration="databricks" module="dagster_databricks" object="PipesDatabricksClient" />**. This method synchronously executes the Databricks job specified by the `SubmitTask` object. It slightly modifies the object by injecting some environment variables under `new_cluster.spark_env_vars` before submitting the object to the Databricks API.

- **Returns a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution**. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the Databricks job has finished. **Note**: Execution can take several minutes even for trivial scripts due to Databricks cluster provisioning times.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> */}
- **Returns a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution**. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the Databricks job has finished. **Note**: Execution can take several minutes even for trivial scripts due to Databricks cluster provisioning times.

### Step 1.2: Define the Databricks Pipes client and Definitions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,8 @@ Here's what we did in this example:

These arguments are passed to the `run` method of <PyObject section="libraries" integration="k8s" module="dagster_k8s" object="PipesK8sClient" />, which submits the provided cluster information to the Kubernetes API and then runs the specified `image`.

- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the execution in Kubernetes has completed.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> */}
- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the execution in Kubernetes has completed.

:::note

Expand Down
3 changes: 1 addition & 2 deletions docs/docs/guides/build/external-resources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,10 @@ Resources allow you to:
## Relevant APIs

{/* TODO replace `ResourceParam` with <PyObject section="resources" module="dagster" object="ResourceParam"/> in table below */}

| Name | Description |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <PyObject section="resources" module="dagster" object="ConfigurableResource"/> | The base class extended to define resources. Under the hood, implements <PyObject section="resources" object="ResourceDefinition" />. |
| `ResourceParam` | An annotation used to specify that a plain Python object parameter for an asset or op is a resource. |
| <PyObject section="resources" module="dagster" object="ResourceParam"/> | An annotation used to specify that a plain Python object parameter for an asset or op is a resource. |
| <PyObject section="resources" module="dagster" object="ResourceDefinition" /> | Class for resource definitions. You almost never want to use initialize this class directly. Instead, you should extend the <PyObject section="resources" object="ConfigurableResource" /> class which implements <PyObject section="resources" object="ResourceDefinition" />. |
| <PyObject section="resources" module="dagster" object="InitResourceContext"/> | The context object provided to a resource during initialization. This object contains required resources, config, and other run information. |
| <PyObject section="resources" module="dagster" object="build_init_resource_context"/> | Function for building an <PyObject section="resources" object="InitResourceContext"/> outside of execution, intended to be used when testing a resource. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,9 @@ import ScaffoldResource from '@site/docs/partials/\_ScaffoldResource.md';

When starting to build a set of assets or jobs, you may want to use a bare Python object without configuration as a resource, such as a third-party API client.

Dagster supports passing bare Python objects as resources. This follows a similar pattern to using a <PyObject section="resources" module="dagster" object="ConfigurableResource"/> subclass; however, assets that use these resources must [annotate](https://docs.python.org/3/library/typing.html#typing.Annotated) them with `ResourceParam`. This annotation lets Dagster know that the parameter is a resource and not an upstream input.

{/* TODO replace `ResourceParam` with <PyObject section="resources" module="dagster" object="ResourceParam"/> */}
Dagster supports passing bare Python objects as resources. This follows a similar pattern to using a <PyObject section="resources" module="dagster" object="ConfigurableResource"/> subclass; however, assets that use these resources must [annotate](https://docs.python.org/3/library/typing.html#typing.Annotated) them with <PyObject section="resources" module="dagster" object="ResourceParam"/>. This annotation lets Dagster know that the parameter is a resource and not an upstream input.

<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource" endBefore="end_raw_github_resource" dedent="4" title="src/<project_name>/defs/assets.py" />

<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource_defs" endBefore="end_raw_github_resource_defs" dedent="4" title="src/<project_name>/defs/resources.py" />
<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource_defs" endBefore="end_raw_github_resource_defs" dedent="4" title="src/<project_name>/defs/resources.py" />
Original file line number Diff line number Diff line change
Expand Up @@ -83,25 +83,8 @@ We recommend using [automation conditions](/guides/automate/declarative-automati

:::


{/* ## Dependencies between time-based partitions and un-partitioned assets */}

{/* TODO */}

{/* ## Dependencies between time-based and static partitions */}

{/* Combining time-based and static partitions allows you to analyze data across both temporal and categorical dimensions. This is particularly useful for scenarios like regional time series analysis. */}

{/* TODO */}

{/* ## Dependencies between time-based and dynamic partitions */}

{/* TODO */}

{/* ## Dependencies between time-based partitions and un-partitioned assets */}

{/* TODO */}

{/* ## Integrating Dagster partitions with external systems: incremental models and dbt */}

{/* TODO */}
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ Sometimes you have a set of predefined categories for your data. For instance, y
<CodeExample path="docs_snippets/docs_snippets/guides/data-modeling/partitioning/static_partitioning.py" language="python" title="src/<project_name>/defs/assets.py"/>

{/* TODO: Link to Backfill page to explain how to backfill regional sales data */}
To backfill historical partitions for regional sales data, see [Backfilling data](/guides/build/partitions-and-backfills/backfilling-data).

## Two-dimensional partitions \{#two-dimensional-partitions}

Expand Down Expand Up @@ -97,4 +98,4 @@ After a partition has been successfully materialized, it will display as green i

To view materializations by partition for a specific asset, navigate to the **Activity** tab of the asset's **Details** page:

![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
Original file line number Diff line number Diff line change
@@ -1,8 +1,39 @@
---
title: Migrating from workspace.yaml to dg.toml
sidebar_position: 2000
description: TODO
# description: TODO
description: Migrate legacy workspace.yaml configurations to a dg.toml workspace.
unlisted: true
---

TODO
TODO

If you have an existing `workspace.yaml` file, you can migrate to a `dg.toml`-based workspace to align with the current Dagster OSS project and workspace structure.

## Overview

`workspace.yaml` directly lists code locations. A `dg.toml` workspace points to one or more Dagster projects, each with its own project configuration. The migration is therefore a move from "direct code locations" to "workspace + projects."

## Step 1: Create a workspace root

If you don't already have a workspace directory, create one by following [Creating workspaces](/guides/build/projects/workspaces/creating-workspaces). This will create a workspace root with a `dg.toml` file.

## Step 2: Convert code locations into projects

Each entry in `workspace.yaml` should map to a Dagster project in your workspace:

1. For each code location, create a Dagster project using [Creating projects](/guides/build/projects/creating-projects).
2. Move or reference your existing definitions in that project.
3. Add each project path under `[[workspace.projects]]` in `dg.toml`.

For the full `dg.toml` format, see the [dg.toml reference](/guides/build/projects/workspaces/dg-toml).

## Step 3: Validate and remove workspace.yaml

Run the workspace from the root directory:

```shell
dg dev
```

If everything loads correctly, you can remove `workspace.yaml` and update any scripts to point to `dg.toml` (for example, `dg dev -w path/to/dg.toml`).
2 changes: 1 addition & 1 deletion docs/docs/guides/test/running-a-subset-of-asset-checks.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ This article assumes familiarity with [asset checks](/guides/test/asset-checks)
Using the <PyObject section="asset-checks" module="dagster" object="multi_asset_check" decorator /> decorator's `specs` and `can_subset` arguments, you can execute a subset of checks in a single op.

{/* TODO link this to proper API doc */}
Inside the body of the function, we can use `AssetCheckExecutionContext.selected_asset_check_keys` to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains.
Inside the body of the function, we can use <PyObject section="execution" module="dagster" object="AssetCheckExecutionContext" /> and its `selected_asset_check_keys` property to identify which computations to run. We can also set the decorator's `can_subset` parameter to `True` to execute a subset of the asset checks that the computation contains.

As we don't know in advance which checks will be executed, we explicitly `yield` each asset check result that we're expected to create:

Expand Down
Loading