Skip to content

Commit 8c44c91

Browse files
Add Documentation about dagster_hf_datasets (#33887)
## Summary & Motivation Since the release of [dagster-hf-datasets](dagster-io/community-integrations#288), this PR adds a documentation page and code examples describing the application and usage of the library. Could you please review the changes? cc: @cmpadden ## Test Plan ## Changelog > The changelog is generated by an agent that examines merged PRs and > summarizes/categorizes user-facing changes. You can optionally replace > this text with a terse description of any user-facing changes in your PR, > which the agent will prioritize. Otherwise, delete this section. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
1 parent 3ce26f0 commit 8c44c91

4 files changed

Lines changed: 351 additions & 0 deletions

File tree

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: Dagster & HF Datasets
3+
sidebar_label: HF Datasets
4+
sidebar_position: 1
5+
description: Use Dagster and Hugging Face Datasets to orchestrate data pipelines, materialize dataset assets, track metadata and lineage, and publish datasets to the Hub.
6+
tags: [community-supported, datasets, metadata, streaming]
7+
source: https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets
8+
pypi: https://pypi.org/project/dagster_hf_datasets/
9+
sidebar_custom_props:
10+
logo: images/integrations/hf_datasets.svg
11+
community: true
12+
partnerlink: https://github.com/huggingface/datasets
13+
canonicalUrl: '/integrations/libraries/hf-datasets'
14+
slug: '/integrations/libraries/hf-datasets'
15+
---
16+
17+
import CommunityIntegration from '@site/docs/partials/\_CommunityIntegration.md';
18+
19+
<CommunityIntegration />
20+
21+
<p>{frontMatter.description}</p>
22+
23+
The integration with HF Datasets makes it easy within Dagster to:
24+
25+
- Load a Hugging Face dataset as a Dagster asset with metadata.
26+
- Stream large datasets efficiently in runtime-only mode
27+
- Capture metadata and lineage for improved observability
28+
- Build multi-asset pipelines that support dataset splits.
29+
- Publish processed datasets back to the Hugging Face Hub.
30+
31+
## Installation
32+
33+
<PackageInstallInstructions packageName="dagster-hf-datasets" />
34+
35+
## Example
36+
37+
This example illustrates the working of a dataset pipeline for transformation, split-aware assets, and Hugging Face Hub publishing.
38+
39+
<CodeExample path="docs_snippets/docs_snippets/integrations/hf_datasets/dataset_pipeline_example.py" language="python" />
40+
41+
## About HF Datasets
42+
43+
[HF Datasets](https://huggingface.co/docs/datasets/index) is a library for accessing, processing, and sharing AI datasets for Audio, Computer Vision, and Natural Language Processing (NLP) tasks.

0 commit comments

Comments
 (0)