Skip to content

README.md

Latest commit

 

History

History
115 lines (73 loc) · 3.71 KB

README.md

File metadata and controls

115 lines (73 loc) · 3.71 KB
Raw

Dagster-HF-Datasets

Dagster-HF-Datasets Logo

Overview

Dagster-HF-Datasets integrates Hugging Face datasets with Dagster for building reproducible, observable data pipelines. Load datasets directly as Dagster assets, apply transformations, and publish results back to the Hub.

Features

  • Hugging Face dataset assets — Load any HF dataset as a Dagster asset with automatic metadata.
  • Streaming support — Efficiently handle large datasets with runtime-only streaming mode.
  • Parquet persistence — Auto-save datasets to disk for caching and versioning.
  • Metadata & lineage — Rich metadata for observability and data lineage tracking.
  • Multi-asset pipelines — Create split-aware assets from datasets with multiple splits.
  • Hub publishing — Push processed datasets back to the Hugging Face Hub with dataset cards.

Installation

pip install dagster-hf-datasets

Development Install:

git clone https://github.com/dagster-io/community-integrations.git

cd libraries/dagster-hf-datasets

pip install -e .

Examples

Basic Asset Pipeline

Get started with a simple example of materializing a Hugging Face dataset as a Dagster asset:

See examples/basic_asset_pipeline.py

  • Dataset materialization with hf_dataset_asset
  • Parquet persistence via HFParquetIOManager
  • Automatic metadata enrichment
  • Hugging Face Hub observability

Multi-Asset Streaming Pipeline

Process large datasets efficiently with runtime-only streaming ingestion:

See examples/multi_asset_pipeline.py

  • Streaming dataset loading with load_dataset(..., streaming=True)
  • Deterministic sampling of IterableDatasets
  • Metadata extraction from streaming sources
  • Conversion to persistent materialized artifacts

Complete Dataset Pipeline

Build production-grade data pipelines with dataset cleaning, transformation and publishing:

See examples/multi_asset_pipeline.py

  • Deduplication and filtering of raw data
  • Text normalization and formatting
  • Multi-step lineage-aware transformations
  • Hugging Face Hub dataset publishing

Documentation

  • Usage Guide — Quick start, configuration, publishing datasets to Hugging Face Hub, and metadata/lineage tracking
  • API Reference — Complete API documentation for HuggingFaceResource, asset decorators, and the IO manager

Resources

Development

Test

make test

Build

make build