Skip to content

Add Documentation about dagster_hf_datasets#33887

Merged
cmpadden merged 10 commits into
dagster-io:masterfrom
ParagEkbote:add-docs-hfdatasets
May 28, 2026
Merged

Add Documentation about dagster_hf_datasets#33887
cmpadden merged 10 commits into
dagster-io:masterfrom
ParagEkbote:add-docs-hfdatasets

Conversation

@ParagEkbote

Copy link
Copy Markdown
Contributor

Summary & Motivation

Since the release of dagster-hf-datasets, this PR adds a documentation page and code examples describing the application and usage of the library. Could you please review the changes?

cc: @cmpadden

Test Plan

Changelog

The changelog is generated by an agent that examines merged PRs and
summarizes/categorizes user-facing changes. You can optionally replace
this text with a terse description of any user-facing changes in your PR,
which the agent will prioritize. Otherwise, delete this section.

@greptile-apps

greptile-apps Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a documentation page and code snippet for the dagster-hf-datasets community integration, which connects Dagster with Hugging Face Datasets for loading, transforming, and publishing datasets.

  • New docs page (docs/docs/integrations/libraries/datasets/index.md) describes installation, capabilities, and links to a pipeline example; the CodeExample path and test-exclusion entry are correctly set after addressing prior review comments.
  • Pipeline snippet (dataset_pipeline_example.py) demonstrates a full multi-asset pipeline: loading the GLUE QQP dataset, deduplication, filtering, normalization, and publishing back to the Hub via HFDatasetPublisher.

Confidence Score: 5/5

Documentation-only change with a companion snippet; no runtime Dagster code paths are modified.

All changed files are docs, a static image, and a test exclusion list. The pipeline snippet is well-structured, MaterializeResult(value=...) is a valid Dagster API, and the test exclusion path correctly includes the hf_datasets/ subdirectory so CI won't attempt to import the snippet without the library installed.

No files require special attention.

Important Files Changed

Filename Overview
docs/docs/integrations/libraries/datasets/index.md New docs page for hf-datasets integration; CodeExample path and slug are correct; minor: pypi URL uses underscores while the rest of the codebase consistently uses hyphens.
examples/docs_snippets/docs_snippets/integrations/hf_datasets/dataset_pipeline_example.py End-to-end pipeline example covering load, dedup, filter, normalize, and publish stages; MaterializeResult(value=...) usage is valid; all assets are wired into Definitions.
examples/docs_snippets/docs_snippets_tests/test_integration_files_load.py Adds hf_datasets/dataset_pipeline_example.py to EXCLUDED_FILES with the correct subdirectory path, preventing CI failures when dagster-hf-datasets is not installed.
docs/static/images/integrations/hf_datasets.svg New SVG logo for the HF Datasets integration sidebar; referenced correctly by the docs page frontmatter.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    HFHub[(Hugging Face Hub
nyu-mll/glue QQP)] -->|hf_dataset_asset decorator| A[raw_glue_qqp
Dataset]
    A -->|HFParquetIOManager| B[deduplicated_glue_qqp
Dataset]
    B -->|HFParquetIOManager| C[filtered_glue_qqp
Dataset]
    C -->|HFParquetIOManager| D[golden_glue_qqp
Dataset]
    D --> E[publish_golden_glue
str: hub URL]
    E -->|HFDatasetPublisher| HFHub2[(Hugging Face Hub
username/golden-glue-qqp)]

    subgraph Resources
        R1[HuggingFaceResource
cache_dir / offline]
        R2[HFParquetIOManager
base_dir]
    end
Loading

Reviews (6): Last reviewed commit: "update wording of doc page." | Re-trigger Greptile

Comment thread docs/docs/integrations/libraries/datasets/index.md Outdated
Comment thread examples/docs_snippets/docs_snippets_tests/test_integration_files_load.py Outdated

@cmpadden cmpadden left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great - thank you @ParagEkbote !

@cmpadden cmpadden added this pull request to the merge queue May 28, 2026
Merged via the queue into dagster-io:master with commit 8c44c91 May 28, 2026
2 of 4 checks passed
elementl-devtools pushed a commit that referenced this pull request May 28, 2026
Fixes the master break from internal build 153552
(https://buildkite.com/dagster/internal/builds/153552):
- prettier/prettier errors in
`dagster-oss/docs/docs/integrations/libraries/datasets/index.md`
introduced by #33887
- ran `prettier --write` against the file using the repo's
`.prettierrc.js`

Generated with Claude Code

Internal-RevId: 36151b4c3ef9a9fa21ab4684518ffd2fbe394b47
@ParagEkbote ParagEkbote deleted the add-docs-hfdatasets branch May 28, 2026 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants