Extend dataset stats: dictionary of DataFrames

[ravi-kumar-pilla]

Parent: #1511

Description

When a dataset loads or saves a dict[str, pd.DataFrame], Kedro Viz does not collect any stats today. We only handle a single DataFrame.

This ticket is about fixing that for the common cases:

• partitions.PartitionedDataset and partitions.IncrementalDataset (with a pandas dataset underneath, e.g. CSV)
• pandas.ExcelDataset when load_args.sheet_name is null and you get all sheets back as a dict

Context

DatasetStatsHook.create_dataset_stats() in package/kedro_viz/integrations/kedro/hooks.py checks isinstance(data, pd.DataFrame) and stops there.

Partitioned datasets and multi sheet Excel both return a dict keyed by partition id or sheet name. The hook still runs on load/save, but nothing gets written to .viz/stats.json.

File size is another gap. get_file_size() expects one filepath. PartitionedDataset uses a directory path instead, so we need a different approach for total size there.

Possible implementation

Add a branch in create_dataset_stats() for dict values where every value is a DataFrame.

Aggregate stats could look like:

• partition or sheet count (len(data))
• total rows (sum of shapes)
• columns (first sheet/partition is probably fine if schemas match, otherwise skip or pick a simple rule and document it)
• file size for Excel via existing filepath logic
• file size for partitions by walking the partition path with fsspec and summing file sizes

Update format_stats() and datasetStatLabels in src/config.js if we expose a new field like partitions or sheets.

Tests in package/tests/test_integrations/test_hooks.py.

Example output:

{
  "partitioned_data": {
    "partitions": 12,
    "rows": 48000,
    "columns": 8,
    "file_size": 1048576
  }
}

Possible alternatives

Show per partition stats in the UI instead of totals. Nice to have but probably a follow up.

Use sheets for Excel and partitions for PartitionedDataset, or one generic label like groups.

Ship rows and counts first and leave partition file size for a later PR if summing files is awkward.

Deliverables

• Hook handles dict[str, pd.DataFrame]
• Stats show up in .viz/stats.json and the metadata panel
• Tests for the dict path
• Short note in RELEASE.md if we add new stat labels users will see

Checklist

• Read CONTRIBUTING.md
• Backend changes in hooks.py
• Frontend label updates if needed
• Tests in test_hooks.py
• Try it manually with PartitionedDataset and/or multi sheet Excel
• Link PR to this issue and Extend Visualize dataset statistics to include additional use cases #1511