Extend dataset stats: Spark datasets

[ravi-kumar-pilla]

Parent: #1511

Description

Spark datasets return a PySpark DataFrame, not pandas. Stats are empty for those today. This ticket covers the main spark kedro-datasets types:

• spark.SparkDataset
• spark.SparkDatasetV2
• spark.SparkHiveDataset
• spark.SparkJDBCDataset

Context

Same hook as the other tickets: DatasetStatsHook in package/kedro_viz/integrations/kedro/hooks.py.

Spark data is lazy and distributed. Getting row count means calling .count(), which kicks off a job. Column count is cheap. Many Spark datasets also have no single local file path, so file size will often stay N/A like it does for SQL backed pandas datasets.

Possible implementation

Detect pyspark.sql.DataFrame with a lazy import.

• columns from len(data.columns)
• rows from data.count() (call out the cost in docs or RELEASE)
• optional partition count from data.rdd.getNumPartitions() if we want another label in the UI
• file size only when we can resolve a filepath on the dataset object

Wrap Spark calls in try/except so tests do not need a live cluster. Mock the DataFrame in unit tests.

If we add partitions as a stat, update format_stats() and datasetStatLabels.

Example:

{
  "model_input@spark": {
    "rows": 1000000,
    "columns": 42,
    "partitions": 8
  }
}

Possible alternatives

Columns only in v1 and skip row count unless the user opts in via settings.

Use table metadata for approximate counts on Hive/Delta instead of a full scan. Harder to do generically.

Bigger refactor with a plugin point for custom stat extractors. Probably overkill for now.

Deliverables

• Hook handles pyspark.sql.DataFrame
• Stats in .viz/stats.json for Spark catalog entries
• Document when row count runs a Spark action and when file size is unavailable
• Unit tests with mocks

Checklist

• Read CONTRIBUTING.md
• Spark branch in create_dataset_stats()
• Decide default for row count (always, opt in, or skip)
• Frontend updates if we add a partitions label
• Mocked tests in test_hooks.py
• Manual check on a Spark project if you have one handy
• Link PR to this issue and Extend Visualize dataset statistics to include additional use cases #1511