From Data Repositories to Production Data Pipelines: Bridging Hugging Face Datasets and Dagster with dagster-hf-datasets

[ParagEkbote]

With the milestone of 1M+ datasets on HF hub, I have contributed an library with hf datasets and dagster with the following features:

• Native support for Dataset and DatasetDict assets
• Automatic lineage tracking and schema observability
• Local dataset materialization with HFParquetIOManager
• Publish curated assets back to the Hub as part of your pipeline

Read the article on HF: https://huggingface.co/blog/AINovice2005/dagster-hf-datasets

Kudos to @cmpadden for collaborating on this release 🙌

[cmpadden]

Very exciting! Thanks for the contribution, @ParagEkbote!

[ParagEkbote]

Published a set of examples covering the use-cases for dagster-hf-datasets, including dataset curation and orchestration patterns: https://huggingface.co/buckets/the-hf-stack/dagster-hf-datasets-examples