DOC-1038 Add dg CLI API docs (#29193)

## Summary & Motivation

Adds `dg` CLI docs to API docs and lightly restructures all API docs:
* Adds Overview section
* Creates sections for Dagster core and Dagster libraries docs for
easier navigation and maintenance -- these sections map to the module
names (`dagster` and `libraries`).
* Moves each section's index file to Sphinx source for easier
maintenance
* Adds a page of CLI documentation to the Labs section


### TODO

- [x] Fix broken links
- [x] Add redirects

## Before

<img width="1185" alt="Screenshot 2025-04-10 at 5 24 05 PM"
src="https://github.com/user-attachments/assets/d42863b9-a131-4930-b4c4-bee5fe219461"
/>

## After

<img width="1213" alt="Screenshot 2025-04-10 at 5 26 34 PM"
src="https://github.com/user-attachments/assets/47810eb1-ddfc-4c02-874b-f19566f5f7ad"
/>

### After with sections collapsed

<img width="1211" alt="Screenshot 2025-04-10 at 5 26 26 PM"
src="https://github.com/user-attachments/assets/c818b4bd-3271-4749-bbf8-8b22b5b5ca1c"
/>


## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <[email protected]>

Author: neverett

docs/CONTRIBUTING.md

@@ -48,7 +48,20 @@ When you make changes to the API, you may need to do some or all of the followin
 * Add or update docstrings in Python files
 * Update reStructuredText files to reference new modules, classes, or methods
 
-In any case, you will need to follow the [reStructuredText syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html).
+#### Formatting API documentation
+
+When updating API docstrings or reStructuredText files, you will need to follow the [reStructuredText syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html). Headings are formatted as follows:
+
+```
+H1 heading
+==========
+
+H2 heading
+----------
+
+H3 heading
+^^^^^^^^^^
+```
 
 ## Formatting
 

docs/docs/api/api-lifecycle/index.md renamed to docs/docs/api/api-lifecycle/api-lifecycle-stages.md

@@ -1,6 +1,6 @@
 ---
-sidebar_class_name: hidden
 title: API lifecycle stages
+sidebar_position: 100
 ---
 
 This reference guide outlines the different stages in the lifecycle of Dagster APIs, from preview to deprecation. Understanding these stages helps developers make informed decisions about which APIs to use in their projects.

docs/docs/api/api-lifecycle/filtering-api-lifecycle-warnings.md

@@ -1,5 +1,6 @@
 ---
 title: Filtering API lifecycle warnings
+sidebar_position: 200
 ---
 
 By default, Dagster logs warnings when APIs marked as preview, beta, superseded and deprecated are used in your code base. These warnings can be filtered out using the `warnings` module available by default in Python.

docs/docs/api/python-api/.gitignore renamed to docs/docs/api/dagster/.gitignore

@@ -1,3 +1,2 @@
 **/*.mdx
-!index.mdx
 !external-assets-rest-api.mdx

docs/docs/api/python-api/external-assets-rest-api.mdx renamed to docs/docs/api/dagster/external-assets-rest-api.mdx

@@ -727,4 +727,4 @@ curl --request POST \
 
 ## Instance API
 
-Refer to the [External assets instance API doc](/api/python-api/external-assets-instance-api) for information on the instance API.
+Refer to the [External assets instance API doc](/api/dagster/external-assets-instance-api) for information on the instance API.

docs/docs/api/index.mdx

@@ -2,6 +2,6 @@
 title: API reference
 ---
 
-These docs aim to cover the entire public surface of the core dagster APIs, as well as public APIs from all provided libraries.
+These docs aim to cover the entire public surface of the core dagster APIs, as well as the dagster-dg API, and public APIs from all provided libraries.
 
 Dagster follows [semantic versioning](https://semver.org/). We attempt to isolate breaking changes to the public APIs to minor versions on a roughly 12-week cadence, and will announce deprecations in Slack and in the release notes to patch versions on a roughly weekly cadence.

docs/docs/api/libraries/.gitignore

@@ -0,0 +1 @@
+**/*.mdx

docs/docs/api/python-api/index.mdx

@@ -19,11 +19,11 @@ Depending on your organization's needs, you may want to retain these logs in you
 
 ## Modifying compute log storage
 
-Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/python-api/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.
+Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/dagster/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.
 
 ### Writing to your own S3 bucket
 
-If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
+If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
 
 You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/management/settings/customizing-agent-settings):
 

docs/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

@@ -98,7 +98,7 @@ compute_logs:
 
 ## Writing compute logs to AWS S3
 
-{/* /api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
+{/* /api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
 You can write compute logs to an AWS S3 bucket by configuring the <PyObject section="libraries" module="dagster_aws" object="s3.S3ComputeLogManager" /> module.
 
 You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.

docs/docs/dagster-plus/deployment/management/settings/customizing-agent-settings.md

@@ -35,7 +35,7 @@ defs = dg.Definitions(
 
 ## 2. Define the DuckDB resource
 
-In Dagster, [Resources](/api/python-api/resources) are the external services, tools, and storage backends you need to do your job. For the storage backend in this project, we'll use [DuckDB](https://duckdb.org/), a fast, in-process SQL database that runs inside your application. We'll define it once in the definitions object, making it available to all assets and objects that need it.
+In Dagster, [Resources](/api/dagster/resources) are the external services, tools, and storage backends you need to do your job. For the storage backend in this project, we'll use [DuckDB](https://duckdb.org/), a fast, in-process SQL database that runs inside your application. We'll define it once in the definitions object, making it available to all assets and objects that need it.
 
 ```python
 defs = dg.Definitions(

docs/docs/etl-pipeline-tutorial/create-and-materialize-assets.md

@@ -12,7 +12,7 @@ To begin we need our specific context. Our RAG system will combine two different
 
 ## GitHub
 
-To retrieve data from GitHub, we are going to borrow code from the [dagster-open-platform](https://github.com/dagster-io/dagster-open-platform). The open platform repository shows how we use Dagster internally, and GitHub is one of the data sources we use, and we wrote a resource to manage pulling that data. The [`GithubResource`](/api/python-api/libraries/dagster-github#resources) allows us to query GitHub using GraphQL. We are most interested in issues and discussions, so our resource will have two methods to retrieve that information over a given date range:
+To retrieve data from GitHub, we are going to borrow code from the [dagster-open-platform](https://github.com/dagster-io/dagster-open-platform). The open platform repository shows how we use Dagster internally, and GitHub is one of the data sources we use, and we wrote a resource to manage pulling that data. The [`GithubResource`](/api/libraries/dagster-github#resources) allows us to query GitHub using GraphQL. We are most interested in issues and discussions, so our resource will have two methods to retrieve that information over a given date range:
 
 <CodeExample
   path="docs_projects/project_ask_ai_dagster/project_ask_ai_dagster/resources/github.py"

docs/docs/examples/rag/sources.md

@@ -9,7 +9,7 @@ With asset selection, you can:
 
 - Select a set of assets to view in the Dagster UI
 - Define a job in Python that targets a selection of assets
-- List or materialize a set of assets using the [Dagster CLI](/api/python-api/cli#dagster-asset)
+- List or materialize a set of assets using the [Dagster CLI](/api/dagster/cli#dagster-asset)
 
 ## Availability
 

docs/docs/guides/build/assets/asset-selection-syntax/index.md

@@ -98,7 +98,7 @@ curl \
 </TabItem>
 </Tabs>
 
-Refer to the [External assets REST API documentation](/api/python-api/external-assets-rest-api) for more information.
+Refer to the [External assets REST API documentation](/api/dagster/external-assets-rest-api) for more information.
 
 ## Modeling a graph of external assets
 

docs/docs/guides/build/assets/external-assets.md

@@ -7,7 +7,7 @@ An **asset** is an object in persistent storage, such as a table, file, or persi
 
 Asset definitions enable a declarative approach to data management, in which code is the source of truth on what data assets should exist and how those assets are computed. To learn how to define assets in code, see "[Defining assets](/guides/build/assets/defining-assets)".
 
-**Materializing** an asset is the act of running its function and saving the results to persistent storage. You can materialize assets from the Dagster UI or by invoking [Python APIs](/api/python-api/).
+**Materializing** an asset is the act of running its function and saving the results to persistent storage. You can materialize assets from the Dagster UI or by invoking [Python APIs](/api/dagster/).
 
 :::note Assets vs ops
 

docs/docs/guides/build/assets/index.md

@@ -6,7 +6,7 @@ sidebar_position: 100
 
 This article covers how to use [Dagster Pipes](/guides/build/external-pipelines/) with [AWS ECS](https://aws.amazon.com/ecs/).
 
-The [dagster-aws](/api/python-api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesECSClient" module="dagster_aws" /> resource which can be used to launch AWS ECS tasks from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the task side.
+The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesECSClient" module="dagster_aws" /> resource which can be used to launch AWS ECS tasks from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the task side.
 
 <details>
   <summary>Prerequisites</summary>

docs/docs/guides/build/external-pipelines/aws/aws-ecs-pipeline.md

@@ -10,7 +10,7 @@ import Preview from '@site/docs/partials/\_Preview.md';
 
 This tutorial gives a short overview on how to use [Dagster Pipes](/guides/build/external-pipelines/) with [AWS EMR on EKS](https://aws.amazon.com/emr/features/eks/) (the corresponding AWS API is called `emr-containers`).
 
-The [dagster-aws](/api/python-api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRContainersClient" module="dagster_aws" /> resource, which can be used to launch EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
+The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRContainersClient" module="dagster_aws" /> resource, which can be used to launch EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
 
 ## Prerequisites
 

docs/docs/guides/build/external-pipelines/aws/aws-emr-containers-pipeline.md

@@ -6,7 +6,7 @@ sidebar_position: 300
 
 This article covers how to use [Dagster Pipes](/guides/build/external-pipelines/) with [AWS EMR](https://aws.amazon.com/emr/).
 
-The [dagster-aws](/api/python-api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
+The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
 
 
 <details>

docs/docs/guides/build/external-pipelines/aws/aws-emr-pipeline.md

@@ -6,7 +6,7 @@ sidebar_position: 400
 
 This article covers how to use [Dagster Pipes](/guides/build/external-pipelines/) with [AWS EMR Serverless](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/getting-started.html).
 
-The [dagster-aws](/api/python-api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRServerlessClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR Serverless jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
+The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesEMRServerlessClient" module="dagster_aws" /> resource, which can be used to launch AWS EMR Serverless jobs from Dagster assets and ops. Dagster can receive regular events such as logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes to your EMR jobs.
 
 
 <details>

docs/docs/guides/build/external-pipelines/aws/aws-emr-serverless-pipeline.md

@@ -8,7 +8,7 @@ sidebar_position: 500
 
 This article covers how to use [Dagster Pipes](/guides/build/external-pipelines/) with [AWS Glue](https://aws.amazon.com/glue/).
 
-The [dagster-aws](/api/python-api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesGlueClient" module="dagster_aws" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
+The [dagster-aws](/api/libraries/dagster-aws) integration library provides the <PyObject section="libraries" object="pipes.PipesGlueClient" module="dagster_aws" /> resource which can be used to launch AWS Glue jobs from Dagster assets and ops. Dagster can receive regular events like logs, asset checks, or asset materializations from jobs launched with this client. Using it requires minimal code changes on the job side.
 
 <details>
     <summary>Prerequisites</summary>

docs/docs/guides/build/external-pipelines/aws/aws-glue-pipeline.md

@@ -98,17 +98,17 @@ The metadata format shown above (`{"raw_value": value, "type": type}`) is part o
 
 Let's review what this code does:
 
-- Imports [`PipesMappingParamsLoader`](/api/python-api/libraries/dagster-pipes#params-loaders) and <PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" /> from `dagster_pipes`
+- Imports [`PipesMappingParamsLoader`](/api/libraries/dagster-pipes#params-loaders) and <PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" /> from `dagster_pipes`
 
 - **Defines a [Lambda function handler](https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html), which is a method in function code that processes events**. This method accepts `event` and `context` arguments, which map to the `event` payload and `context` we'll define in our Dagster asset.
 
 - **Initializes the Dagster Pipes context (<PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" />), which yields an instance of <PyObject section="libraries" object="PipesContext" module="dagster_pipes" /> called `pipes`.**
 
-  On the orchestration side - which we'll discuss in the next section - we'll set up a Dagster asset that uses the <PyObject section="libraries" module="dagster_aws" object="pipes.PipesLambdaClient" /> to inject information needed for Pipes in an `event` payload. In this code on the AWS Lambda side, we're passing this payload to [`PipesMappingParamsLoader`](/api/python-api/libraries/dagster-pipes#params-loaders) and using it in <PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" />.
+  On the orchestration side - which we'll discuss in the next section - we'll set up a Dagster asset that uses the <PyObject section="libraries" module="dagster_aws" object="pipes.PipesLambdaClient" /> to inject information needed for Pipes in an `event` payload. In this code on the AWS Lambda side, we're passing this payload to [`PipesMappingParamsLoader`](/api/libraries/dagster-pipes#params-loaders) and using it in <PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" />.
 
   We're using the default context loader (<PyObject section="libraries" object="PipesDefaultContextLoader" module="dagster_pipes" />) and message writer (<PyObject section="libraries" object="PipesDefaultMessageWriter" module="dagster_pipes" />) in this example. These objects establish communication between the orchestration and external process. On the orchestration end, these match a corresponding `PipesLambdaEventContextInjector` and `PipesLambdaLogsMessageReader`, which are instantiated inside the <PyObject section="libraries" module="dagster_aws" object="pipes.PipesLambdaClient" />.
 
-- **Inside the body of the context manager (<PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" />), retrieve a log and report an asset materialization.** These calls use the temporary communications channels established by <PyObject section="libraries" object="PipesDefaultContextLoader" module="dagster_pipes" /> and <PyObject section="libraries" object="PipesDefaultMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject section="libraries" object="PipesContext" module="dagster_pipes" />, see the [API docs](/api/python-api/pipes) or the general [Pipes documentation](/guides/build/external-pipelines).
+- **Inside the body of the context manager (<PyObject section="libraries" object="open_dagster_pipes" module="dagster_pipes" />), retrieve a log and report an asset materialization.** These calls use the temporary communications channels established by <PyObject section="libraries" object="PipesDefaultContextLoader" module="dagster_pipes" /> and <PyObject section="libraries" object="PipesDefaultMessageWriter" module="dagster_pipes" />. To see the full range of what you can do with the <PyObject section="libraries" object="PipesContext" module="dagster_pipes" />, see the [API docs](/api/dagster/pipes) or the general [Pipes documentation](/guides/build/external-pipelines).
 
 At this point you can execute the rest of your AWS Lambda code as normal, invoking various <PyObject section="libraries" object="PipesContext" module="dagster_pipes" /> APIs as needed.