Skip to content

Commit 43e1076

Browse files
committed
update integrations API docs frontmatter and links to API integrations docs
Signed-off-by: nikki everett <nikki@dagsterlabs.com>
1 parent aa6d2a2 commit 43e1076

113 files changed

Lines changed: 226 additions & 123 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/docs/deployment/dagster-plus/management/customizing-agent-settings.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ compute_logs:
9696

9797
## Writing compute logs to AWS S3
9898

99-
{/* /api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
99+
{/* /integrations/libraries/aws/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
100100
You can write compute logs to an AWS S3 bucket by configuring the <PyObject section="libraries" module="dagster_aws" object="s3.S3ComputeLogManager" /> module.
101101

102102
You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.

docs/docs/deployment/dagster-plus/management/managing-compute-logs-and-error-messages.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/
1919

2020
### Writing to your own S3 bucket
2121

22-
If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
22+
If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/integrations/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
2323

2424
You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/deployment/dagster-plus/management/customizing-agent-settings):
2525

docs/docs/deployment/execution/run-monitoring.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,8 +76,8 @@ If a run worker crashes, the run it's managing can hang. The monitoring daemon c
7676
7777
This feature is currently only supported when using:
7878
79-
- [`K8sRunLauncher`](/api/libraries/dagster-k8s#dagster_k8s.K8sRunLauncher) with the [`k8s_job_executor`](/api/libraries/dagster-k8s#dagster_k8s.k8s_job_executor)
80-
- [`DockerRunLauncher`](/api/libraries/dagster-docker#dagster_docker.DockerRunLauncher) with the [`docker_executor`](/api/libraries/dagster-docker#dagster_docker.docker_executor)
79+
- [`K8sRunLauncher`](/integrations/libraries/dagster-k8s#dagster_k8s.K8sRunLauncher) with the [`k8s_job_executor`](/integrations/libraries/dagster-k8s#dagster_k8s.k8s_job_executor)
80+
- [`DockerRunLauncher`](/integrations/libraries/dagster-docker#dagster_docker.DockerRunLauncher) with the [`docker_executor`](/integrations/libraries/dagster-docker#dagster_docker.docker_executor)
8181

8282
The monitoring daemon handles these by performing health checks on the run workers. If a failure is detected, the daemon can launch a new run worker which resumes execution of the existing run. The run worker crash will be show in the event log, and the run will continue to completion. If the run worker continues to crash, the daemon will mark the run as failed after the configured number of attempts.
8383

docs/docs/deployment/oss/oss-instance-configuration.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -100,11 +100,11 @@ To use a SQLite database for storage, configure `storage.sqlite` in `dagster.yam
100100

101101
:::note
102102

103-
To use Postgres storage, you'll need to install the [dagster-postgres](/api/libraries/dagster-postgres) library.
103+
To use Postgres storage, you'll need to install the [dagster-postgres](/integrations/libraries/dagster-postgres) library.
104104

105105
:::
106106

107-
To use a [PostgreSQL database](/api/libraries/dagster-postgres) for storage, configure `storage.postgres` in `dagster.yaml`:
107+
To use a [PostgreSQL database](/integrations/libraries/dagster-postgres) for storage, configure `storage.postgres` in `dagster.yaml`:
108108

109109
<CodeExample
110110
path="docs_snippets/docs_snippets/deploying/dagster_instance/dagster.yaml"
@@ -119,11 +119,11 @@ To use a [PostgreSQL database](/api/libraries/dagster-postgres) for storage, con
119119

120120
:::note
121121

122-
To use MySQL storage, you'll need to install the [dagster-mysql](/api/libraries/dagster-mysql) library.
122+
To use MySQL storage, you'll need to install the [dagster-mysql](/integrations/libraries/dagster-mysql) library.
123123

124124
:::
125125

126-
To use a [MySQL database](/api/libraries/dagster-mysql) for storage, configure `storage.mysql` in `dagster.yaml`:
126+
To use a [MySQL database](/integrations/libraries/dagster-mysql) for storage, configure `storage.mysql` in `dagster.yaml`:
127127

128128
<CodeExample
129129
path="docs_snippets/docs_snippets/deploying/dagster_instance/dagster.yaml"

docs/docs/examples/full-pipelines/rag/sources.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ To begin we need our specific context. Our RAG system will combine two different
1212

1313
## GitHub
1414

15-
To retrieve data from GitHub, we are going to borrow code from the [dagster-open-platform](https://github.com/dagster-io/dagster-open-platform). The open platform repository shows how we use Dagster internally, and GitHub is one of the data sources we use, and we wrote a resource to manage pulling that data. The [`GithubResource`](/api/libraries/dagster-github#resources) allows us to query GitHub using GraphQL. We are most interested in issues and discussions, so our resource will have two methods to retrieve that information over a given date range:
15+
To retrieve data from GitHub, we are going to borrow code from the [dagster-open-platform](https://github.com/dagster-io/dagster-open-platform). The open platform repository shows how we use Dagster internally, and GitHub is one of the data sources we use, and we wrote a resource to manage pulling that data. The [`GithubResource`](/integrations/libraries/dagster-github#resources) allows us to query GitHub using GraphQL. We are most interested in issues and discussions, so our resource will have two methods to retrieve that information over a given date range:
1616

1717
<CodeExample
1818
path="docs_projects/project_ask_ai_dagster/src/project_ask_ai_dagster/defs/github.py"

docs/docs/guides/build/external-pipelines/aws/aws-ecs-pipeline.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import ScaffoldProject from '@site/docs/partials/\_ScaffoldProject.md';
88

99
This article covers how to use [Dagster Pipes](/guides/build/external-pipelines) with [AWS ECS](https://aws.amazon.com/ecs).
1010

11-
The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesECSClient" module="dagster_aws" /> resource which can be used to launch AWS ECS tasks from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the task side.
11+
The [dagster-aws](/integrations/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesECSClient" module="dagster_aws" /> resource which can be used to launch AWS ECS tasks from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the task side.
1212

1313
## Prerequisites
1414

docs/docs/guides/build/external-pipelines/aws/aws-emr-containers-pipeline.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ import ScaffoldProject from '@site/docs/partials/\_ScaffoldProject.md';
1111

1212
This tutorial gives a short overview on how to use [Dagster Pipes](/guides/build/external-pipelines) with [AWS EMR on EKS](https://aws.amazon.com/emr/features/eks) (the corresponding AWS API is called `emr-containers`).
1313

14-
The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRContainersClient" module="dagster_aws" /> resource, which can be used to launch EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
14+
The [dagster-aws](/integrations/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRContainersClient" module="dagster_aws" /> resource, which can be used to launch EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
1515

1616
## Prerequisites
1717

docs/docs/guides/build/external-pipelines/aws/aws-emr-pipeline.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import ScaffoldProject from '@site/docs/partials/\_ScaffoldProject.md';
88

99
This article covers how to use [Dagster Pipes](/guides/build/external-pipelines) with [AWS EMR](https://aws.amazon.com/emr).
1010

11-
The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
11+
The [dagster-aws](/integrations/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
1212

1313

1414
## Prerequisites

docs/docs/guides/build/external-pipelines/aws/aws-emr-serverless-pipeline.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import ScaffoldProject from '@site/docs/partials/\_ScaffoldProject.md';
88

99
This article covers how to use [Dagster Pipes](/guides/build/external-pipelines) with [AWS EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/getting-started.html).
1010

11-
The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRServerlessClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR Serverless jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
11+
The [dagster-aws](/integrations/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRServerlessClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR Serverless jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
1212

1313
## Prerequisites
1414

docs/docs/guides/build/external-pipelines/aws/aws-glue-pipeline.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ import ScaffoldProject from '@site/docs/partials/\_ScaffoldProject.md';
1010

1111
This article covers how to use [Dagster Pipes](/guides/build/external-pipelines) with [AWS Glue](https://aws.amazon.com/glue).
1212

13-
The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesGlueClient" module="dagster_aws" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
13+
The [dagster-aws](/integrations/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesGlueClient" module="dagster_aws" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
1414

1515
## Prerequisites
1616

0 commit comments

Comments
 (0)