DOC-1038 Add dg CLI API docs (#29193)

## Summary & Motivation

Adds `dg` CLI docs to API docs and lightly restructures all API docs:
* Adds Overview section
* Creates sections for Dagster core and Dagster libraries docs for
easier navigation and maintenance -- these sections map to the module
names (`dagster` and `libraries`).
* Moves each section's index file to Sphinx source for easier
maintenance
* Adds a page of CLI documentation to the Labs section


### TODO

- [x] Fix broken links
- [x] Add redirects

## Before

<img width="1185" alt="Screenshot 2025-04-10 at 5 24 05 PM"
src="https://github.com/user-attachments/assets/d42863b9-a131-4930-b4c4-bee5fe219461"
/>

## After

<img width="1213" alt="Screenshot 2025-04-10 at 5 26 34 PM"
src="https://github.com/user-attachments/assets/47810eb1-ddfc-4c02-874b-f19566f5f7ad"
/>

### After with sections collapsed

<img width="1211" alt="Screenshot 2025-04-10 at 5 26 26 PM"
src="https://github.com/user-attachments/assets/c818b4bd-3271-4749-bbf8-8b22b5b5ca1c"
/>


## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.

---------

Signed-off-by: nikki everett <[email protected]>

Author: neverett

docs/CONTRIBUTING.md

@@ -48,7 +48,20 @@ When you make changes to the API, you may need to do some or all of the followin
 * Add or update docstrings in Python files
 * Update reStructuredText files to reference new modules, classes, or methods
 
-In any case, you will need to follow the [reStructuredText syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html).
+#### Formatting API documentation
+
+When updating API docstrings or reStructuredText files, you will need to follow the [reStructuredText syntax](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html). Headings are formatted as follows:
+
+```
+H1 heading
+==========
+
+H2 heading
+----------
+
+H3 heading
+^^^^^^^^^^
+```
 
 ## Formatting
 

docs/docs/api/api-lifecycle/index.md renamed to docs/docs/api/api-lifecycle/api-lifecycle-stages.md

@@ -1,6 +1,6 @@
 ---
-sidebar_class_name: hidden
 title: API lifecycle stages
+sidebar_position: 100
 ---
 
 This reference guide outlines the different stages in the lifecycle of Dagster APIs, from preview to deprecation. Understanding these stages helps developers make informed decisions about which APIs to use in their projects.

docs/docs/api/api-lifecycle/filtering-api-lifecycle-warnings.md

@@ -1,5 +1,6 @@
 ---
 title: Filtering API lifecycle warnings
+sidebar_position: 200
 ---
 
 By default, Dagster logs warnings when APIs marked as preview, beta, superseded and deprecated are used in your code base. These warnings can be filtered out using the `warnings` module available by default in Python.

docs/docs/api/python-api/.gitignore renamed to docs/docs/api/dagster/.gitignore

@@ -1,3 +1,2 @@
 **/*.mdx
-!index.mdx
 !external-assets-rest-api.mdx

docs/docs/api/python-api/external-assets-rest-api.mdx renamed to docs/docs/api/dagster/external-assets-rest-api.mdx

@@ -727,4 +727,4 @@ curl --request POST \
 
 ## Instance API
 
-Refer to the [External assets instance API doc](/api/python-api/external-assets-instance-api) for information on the instance API.
+Refer to the [External assets instance API doc](/api/dagster/external-assets-instance-api) for information on the instance API.

docs/docs/api/index.mdx

@@ -2,6 +2,6 @@
 title: API reference
 ---
 
-These docs aim to cover the entire public surface of the core dagster APIs, as well as public APIs from all provided libraries.
+These docs aim to cover the entire public surface of the core dagster APIs, as well as the dagster-dg API, and public APIs from all provided libraries.
 
 Dagster follows [semantic versioning](https://semver.org/). We attempt to isolate breaking changes to the public APIs to minor versions on a roughly 12-week cadence, and will announce deprecations in Slack and in the release notes to patch versions on a roughly weekly cadence.

docs/docs/api/libraries/.gitignore

@@ -0,0 +1 @@
+**/*.mdx

docs/docs/api/python-api/index.mdx

@@ -19,11 +19,11 @@ Depending on your organization's needs, you may want to retain these logs in you
 
 ## Modifying compute log storage
 
-Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/python-api/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.
+Dagster's compute logs are handled by the configured [`ComputeLogManager`](/api/dagster/internals#compute-log-manager). By default, Dagster+ utilizes the `CloudComputeLogManager` which stores logs in a Dagster+-managed Amazon S3 bucket, but you can customize this behavior to store logs in a destination of your choice.
 
 ### Writing to your own S3 bucket
 
-If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
+If using the Kubernetes agent, you can instead forward logs to your own S3 bucket by using the [`S3ComputeLogManager`](/api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager).
 
 You can configure the `S3ComputeLogManager` in your [`dagster.yaml` file](/dagster-plus/deployment/management/settings/customizing-agent-settings):
 

docs/docs/dagster-plus/deployment/management/managing-compute-logs-and-error-messages.md

@@ -98,7 +98,7 @@ compute_logs:
 
 ## Writing compute logs to AWS S3
 
-{/* /api/python-api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
+{/* /api/libraries/dagster-aws#dagster_aws.s3.S3ComputeLogManager */}
 You can write compute logs to an AWS S3 bucket by configuring the <PyObject section="libraries" module="dagster_aws" object="s3.S3ComputeLogManager" /> module.
 
 You are also able to stream partial compute log files by configuring the log upload interval (in seconds) using the `upload_interval` parameter.