[dagster-spark] mark spark declarative pipelines as preview; add API doc references

[cmpadden]

Summary & Motivation

Follow up to #33539

• Marks user-facing APIs as preview
• Adds references to docs RST files

Test Plan

Changelog

• [dagster-spark] Introduces Spark Declarative Pipeline support in feature preview

[greptile-apps]

Greptile Summary

This PR marks all three user-facing Spark Declarative Pipeline APIs — SparkDeclarativePipelineComponent, SparkPipelinesResource, and SparkDeclarativePipelineScaffolder — as preview using the standard dagster._annotations.preview decorator, and adds the corresponding API reference entries to the Sphinx documentation.

Key changes:

• @preview added to all three user-facing classes following the established pattern used in other Dagster integrations (e.g., dagster-gcp's BigQueryResourceComponent).
• RST documentation correctly uses .. currentmodule:: scoping: SparkDeclarativePipelineComponent is documented under dagster_spark (where it is re-exported in __init__.py), while SparkPipelinesResource and SparkDeclarativePipelineScaffolder use dagster_spark.components.spark_declarative_pipeline.
• Both test files add a module-level pytestmark to suppress dagster.PreviewWarning, which is the correct way to silence expected preview warnings in test suites.

Confidence Score: 5/5

• This PR is safe to merge — the changes are minimal, mechanical, and follow existing patterns across the codebase.
• All three changes are additive (decorator + test warning suppression + RST docs). Module scoping in the RST file is correct relative to what's exported in __init__.py. The @preview pattern on component classes and configurable resources is already established in other Dagster integrations. Tests continue to cover the same scenarios with the only change being warning suppression.
• No files require special attention.

Important Files Changed

Filename	Overview
python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/component.py	Adds @preview decorator above @scaffold_with and @dataclass on SparkDeclarativePipelineComponent; correctly follows the existing pattern used in other Dagster library components.
python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/resource.py	Adds @preview decorator to SparkPipelinesResource (a ConfigurableResource subclass); clean, minimal change.
python_modules/libraries/dagster-spark/dagster_spark/components/spark_declarative_pipeline/scaffolder.py	Adds @preview decorator to SparkDeclarativePipelineScaffolder; clean change consistent with the other two classes in this PR.
docs/sphinx/sections/integrations/libraries/spark/dagster-spark.rst	Adds a new "Spark Declarative Pipelines" section with autoclass directives for all three new preview classes; currentmodule is correctly scoped — SparkDeclarativePipelineComponent is found under dagster_spark (re-exported in __init__.py) while SparkPipelinesResource and SparkDeclarativePipelineScaffolder use dagster_spark.components.spark_declarative_pipeline.
python_modules/libraries/dagster-spark/dagster_spark_tests/components/spark_declarative_pipeline/test_component.py	Adds module-level pytestmark to suppress dagster.PreviewWarning across all component tests; correct approach given the new @preview decorators.
python_modules/libraries/dagster-spark/dagster_spark_tests/components/spark_declarative_pipeline/test_resource.py	Adds module-level pytestmark to suppress dagster.PreviewWarning across all resource tests; pytest was already imported so no additional import needed.

Sequence Diagram

sequenceDiagram
    participant User
    participant Component as SparkDeclarativePipelineComponent<br/>(@preview)
    participant Resource as SparkPipelinesResource<br/>(@preview)
    participant Scaffolder as SparkDeclarativePipelineScaffolder<br/>(@preview)
    participant CLI as spark-pipelines CLI

    User->>Scaffolder: scaffold(request)
    Note over Scaffolder: Emits PreviewWarning
    Scaffolder-->>User: defs.yaml + pipeline spec

    User->>Component: load from YAML
    Note over Component: Emits PreviewWarning
    Component->>CLI: dry-run / source_only discovery
    CLI-->>Component: discovered datasets
    Component->>Component: write_state_to_path()

    User->>Component: build_defs_from_state()
    Component->>Component: deserialize SparkPipelineState
    Component->>Component: filter temporary_view datasets
    Component-->>User: Definitions (multi_asset + resource)

    User->>Resource: run_and_observe(context, spec_path)
    Note over Resource: Emits PreviewWarning
    Resource->>CLI: spark-pipelines run --spec ...
    CLI-->>Resource: stdout/stderr (streaming)
    Resource->>Resource: context.log.info(line)
    Resource-->>User: raises SparkPipelinesExecutionError on non-zero exit
    User->>User: yield MaterializeResult per asset key

Loading

_{Reviews (1): Last reviewed commit: "[dagster-spark] mark spark declarative p..." | Re-trigger Greptile}