Skip to content
Closed
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ To override a code location role for an individual user:
2. Click **Edit**.
3. Click the toggle to the left of the deployment to open a list of code locations.
4. Next to a code location, click **Edit user role**.
5. Select the user role for the code location: {/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override.png" */}
5. Select the user role for the code location.
6. Click **Save**.

#### Team members
Expand All @@ -118,13 +118,9 @@ The above also applies to code locations and Branch Deployment roles.

#### Viewing overrides

To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment:
To view deployment-level overrides for a specific user, locate the user on the **Users** page and hover over a deployment. A tooltip will display any overrides that apply at the deployment level.

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/user-overrides-popup.png" */}

If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides:

{/* TODO: add picture previously at "/images/dagster-cloud/user-token-management/code-location-override-popup.png" */}
If there are code location-level overrides, a small **N override(s)** link will display beneath the user's deployment role. Hover over it to display the list of overrides.

#### Removing overrides

Expand Down
4 changes: 1 addition & 3 deletions docs/docs/deployment/dagster-plus/hybrid/kubernetes/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,7 @@ helm --namespace dagster-cloud upgrade agent \

## Troubleshooting tips

You can see basic health information about your agent in the Dagster+ UI:

[comment]: <> (TODO: Screenshot of Dagster+ Deployments agents tab)
You can see basic health information about your agent in the Dagster+ UI.

### View logs

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,7 @@ Now copy the following sensor code in the `sensors.py` file:
2. Navigate to the Automation page.
3. Turn on the `adhoc_request_sensor`.
4. Click on the `adhoc_request_sensor` details.

{/* TODO: Screenshot */}
You should see the sensor details view with recent ticks and the run history panel.

5. Create a `data/requests` directory in `dagster_tutorial`. Then include a `request.json` file:

Expand All @@ -76,8 +75,7 @@ Now copy the following sensor code in the `sensors.py` file:
```

6. Click on the green tick to see the run for this request.

{/* TODO: Screenshot */}
The run should appear in the runs list, and the tick details will show the run ID.

## Summary

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,7 @@ Then we can execute this job and pull the attributes we need to build the `conte
title="src/<project_name>/defs/sensors.py"
/>

{/* TODO the methods and statuses below do not exist in API docs */}
{/* We have provided convenience functions <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_success_event" /> and <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" method="get_job_failure_event" /> for retrieving `DagsterRunStatus.SUCCESS` and `DagsterRunStatus.FAILURE` events, respectively. If you have a run status sensor triggered on another status, you can retrieve all events from `result` and filter based on your event type. */}
To inspect the resulting status in tests, use the <PyObject section="execution" module="dagster" object="ExecuteInProcessResult" /> returned by `execute_in_process`. Its `dagster_run` and `all_events` properties let you filter for the status or event type you care about.

We can use the same pattern to build the context for <PyObject section="schedules-sensors" module="dagster" object="run_failure_sensor" />. If we wanted to test this run failure sensor:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@ Dagster's versioning system helps you determine ahead of time whether materializ
Dagster has two versioning concepts to represent the code and input data used for each materialization:

- **Code version.** A string that represents the version of the code that computes an asset. This is the `code_version` argument of <PyObject section="assets" module="dagster" object="asset" decorator />.
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a `DataVersion` object.
{/* TODO link `DataVersion` to API docs */}
- **Data version.** A string that represents the version of the data represented by the asset. This is represented as a <PyObject section="assets" module="dagster" object="DataVersion" /> object.

By keeping track of code and data versions, Dagster can predict whether a materialization will change the underlying value. This allows Dagster to skip redundant materializations and instead return the previously computed value. In more technical terms, Dagster offers a limited form of [memoization](https://en.wikipedia.org/wiki/Memoization) for assets: the last-computed asset value is always cached.

Expand Down
5 changes: 4 additions & 1 deletion docs/docs/guides/build/assets/passing-data-between-assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,7 @@ The downsides of this approach are:

## Related resources

{/* TODO: add links to relevant API documentation here. */}
- <PyObject section="assets" module="dagster" object="AssetIn" />
- <PyObject section="assets" module="dagster" object="AssetOut" />
- <PyObject section="io-managers" module="dagster" object="IOManager" />
- <PyObject section="execution" module="dagster" object="AssetExecutionContext" />
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,18 @@ sidebar_position: 500
unlisted: true
---

TODO
This page lists common checks when a component fails to load or behaves unexpectedly.

## Quick checks

- **Validate the component type string**. Ensure the type name matches the registered component class and that the package providing it is installed in your environment.
- **Confirm configuration fields**. Component configuration is validated on load. If a field is missing or mis-typed, the error message will include the field path to update.
- **Verify workspace paths**. If you are using a workspace, confirm that `dg.toml` points at the correct project directories and that the project loads on its own.

## Inspecting load errors

If a component fails to load, the code location will show a load error in the UI and in CLI output. Use the stack trace there to locate the file and configuration entry that caused the failure.

## Next steps

For a refresher on how components are structured, see the [components overview](/guides/build/components).
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,7 @@ Here's what we did in this example:

This argument is passed to the `run` method of <PyObject section="libraries" integration="aws" module="dagster_aws" object="pipes.PipesLambdaClient" />, which submits the provided information to the [boto `invoke` API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda/client/invoke.html) and then invokes the specified function (`function_name`).

- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the execution in AWS Lambda has completed.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="libraries" module="pipes" object="dagster.PipesClientCompletedInvocation" /> */}
- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the execution in AWS Lambda has completed.
Comment thread
Vamsi-klu marked this conversation as resolved.

### Step 2.2: Create Dagster Definitions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,8 +95,7 @@ Let's review what's happening in this code:

- **Passes the `SubmitTask` object, `AssetExecutionContext`, and `extras` dictionary to the `run` method of <PyObject section="libraries" integration="databricks" module="dagster_databricks" object="PipesDatabricksClient" />**. This method synchronously executes the Databricks job specified by the `SubmitTask` object. It slightly modifies the object by injecting some environment variables under `new_cluster.spark_env_vars` before submitting the object to the Databricks API.

- **Returns a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution**. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the Databricks job has finished. **Note**: Execution can take several minutes even for trivial scripts due to Databricks cluster provisioning times.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> */}
- **Returns a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution**. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the Databricks job has finished. **Note**: Execution can take several minutes even for trivial scripts due to Databricks cluster provisioning times.

### Step 1.2: Define the Databricks Pipes client and Definitions

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,8 +164,7 @@ Here's what we did in this example:

These arguments are passed to the `run` method of <PyObject section="libraries" integration="k8s" module="dagster_k8s" object="PipesK8sClient" />, which submits the provided cluster information to the Kubernetes API and then runs the specified `image`.

- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the `PipesClientCompletedInvocation` object returned by `run` after the execution in Kubernetes has completed.
{/* TODO replace `PipesClientCompletedInvocation` with <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> */}
- Returned a <PyObject section="assets" module="dagster" object="MaterializeResult" /> object representing the result of execution. This is obtained by calling `get_materialize_result` on the <PyObject section="pipes" module="dagster" object="PipesClientCompletedInvocation" /> object returned by `run` after the execution in Kubernetes has completed.

:::note

Expand Down
4 changes: 1 addition & 3 deletions docs/docs/guides/build/external-resources/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,10 @@ Resources allow you to:

## Relevant APIs

{/* TODO replace `ResourceParam` with <PyObject section="resources" module="dagster" object="ResourceParam"/> in table below */}

| Name | Description |
| ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <PyObject section="resources" module="dagster" object="ConfigurableResource"/> | The base class extended to define resources. Under the hood, implements <PyObject section="resources" object="ResourceDefinition" />. |
| `ResourceParam` | An annotation used to specify that a plain Python object parameter for an asset or op is a resource. |
| <PyObject section="resources" module="dagster" object="ResourceParam"/> | An annotation used to specify that a plain Python object parameter for an asset or op is a resource. |
| <PyObject section="resources" module="dagster" object="ResourceDefinition" /> | Class for resource definitions. You almost never want to use initialize this class directly. Instead, you should extend the <PyObject section="resources" object="ConfigurableResource" /> class which implements <PyObject section="resources" object="ResourceDefinition" />. |
| <PyObject section="resources" module="dagster" object="InitResourceContext"/> | The context object provided to a resource during initialization. This object contains required resources, config, and other run information. |
| <PyObject section="resources" module="dagster" object="build_init_resource_context"/> | Function for building an <PyObject section="resources" object="InitResourceContext"/> outside of execution, intended to be used when testing a resource. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ import ScaffoldResource from '@site/docs/partials/\_ScaffoldResource.md';

When starting to build a set of assets or jobs, you may want to use a bare Python object without configuration as a resource, such as a third-party API client.

Dagster supports passing bare Python objects as resources. This follows a similar pattern to using a <PyObject section="resources" module="dagster" object="ConfigurableResource"/> subclass; however, assets that use these resources must [annotate](https://docs.python.org/3/library/typing.html#typing.Annotated) them with `ResourceParam`. This annotation lets Dagster know that the parameter is a resource and not an upstream input.

{/* TODO replace `ResourceParam` with <PyObject section="resources" module="dagster" object="ResourceParam"/> */}
Dagster supports passing bare Python objects as resources. This follows a similar pattern to using a <PyObject section="resources" module="dagster" object="ConfigurableResource"/> subclass; however, assets that use these resources must [annotate](https://docs.python.org/3/library/typing.html#typing.Annotated) them with <PyObject section="resources" module="dagster" object="ResourceParam"/>. This annotation lets Dagster know that the parameter is a resource and not an upstream input.

<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource" endBefore="end_raw_github_resource" dedent="4" title="src/<project_name>/defs/assets.py" />

<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource_defs" endBefore="end_raw_github_resource_defs" dedent="4" title="src/<project_name>/defs/resources.py" />
<CodeExample path="docs_snippets/docs_snippets/concepts/resources/pythonic_resources.py" startAfter="start_raw_github_resource_defs" endBefore="end_raw_github_resource_defs" dedent="4" title="src/<project_name>/defs/resources.py" />
Original file line number Diff line number Diff line change
Expand Up @@ -83,25 +83,3 @@ We recommend using [automation conditions](/guides/automate/declarative-automati

:::


{/* ## Dependencies between time-based partitions and un-partitioned assets */}

{/* TODO */}

{/* ## Dependencies between time-based and static partitions */}

{/* Combining time-based and static partitions allows you to analyze data across both temporal and categorical dimensions. This is particularly useful for scenarios like regional time series analysis. */}

{/* TODO */}

{/* ## Dependencies between time-based and dynamic partitions */}

{/* TODO */}

{/* ## Dependencies between time-based partitions and un-partitioned assets */}

{/* TODO */}

{/* ## Integrating Dagster partitions with external systems: incremental models and dbt */}

{/* TODO */}
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Sometimes you have a set of predefined categories for your data. For instance, y

<CodeExample path="docs_snippets/docs_snippets/guides/data-modeling/partitioning/static_partitioning.py" language="python" title="src/<project_name>/defs/assets.py"/>

{/* TODO: Link to Backfill page to explain how to backfill regional sales data */}
To backfill historical partitions for regional sales data, see [Backfilling data](/guides/build/partitions-and-backfills/backfilling-data).

## Two-dimensional partitions \{#two-dimensional-partitions}

Expand Down Expand Up @@ -97,4 +97,4 @@ After a partition has been successfully materialized, it will display as green i

To view materializations by partition for a specific asset, navigate to the **Activity** tab of the asset's **Details** page:

![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
![Asset activity section of asset details page](/images/guides/build/partitions-and-backfills/materialized-partitioned-asset-activity.png)
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ Dagster+ automatically tracks metadata every time a project (code location) is l
1. In the Dagster+ UI, navigate to the **Deployment** tab.
2. In the row associated with the project you're interested in, click **View history** in the **Updated** column.

{/* TODO - update screenshot when code location renamed in UI */}
![Screenshot highlighting the "Updated" column for a project/code location](/images/dagster-plus/deployment/code-locations/view-code-location-history.png)

This will bring up a modal showing a history of every time that project has been loaded, and metadata associated with that load. If you have connected Dagster+ to a GitHub or GitLab repository, each row will have a link to the commit that was deployed at that point in time.
Expand Down Expand Up @@ -45,7 +44,6 @@ If you notice an issue with newly deployed code, or your code fails to deploy su
2. In the row associated with the project you're interested in, click **View history** in the **Updated** column.
3. In the **Actions** column click the dropdown menu to the right of **View metadata**, select **Rollback to this version**.

{/* TODO - update screenshot when code location renamed in UI */}
![Screenshot highlighting the "Updated" column for a project/code location](/images/dagster-plus/deployment/code-locations/rollback-code-location.png)

:::tip
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,5 +99,4 @@ Finally, let's load our two projects with `dg dev`. When you run `dg dev` from t

<CliInvocationExample contents="dg dev" />

{/* TODO - replace screenshot */}
![](/images/guides/build/projects-and-components/setting-up-a-workspace/two-projects.png)
Loading