Add Documentation about dagster_hf_datasets#33887
Conversation
…dagster into add-docs-hfdatasets
Greptile SummaryThis PR adds a documentation page and code snippet for the
Confidence Score: 5/5Documentation-only change with a companion snippet; no runtime Dagster code paths are modified. All changed files are docs, a static image, and a test exclusion list. The pipeline snippet is well-structured, MaterializeResult(value=...) is a valid Dagster API, and the test exclusion path correctly includes the hf_datasets/ subdirectory so CI won't attempt to import the snippet without the library installed. No files require special attention.
|
| Filename | Overview |
|---|---|
| docs/docs/integrations/libraries/datasets/index.md | New docs page for hf-datasets integration; CodeExample path and slug are correct; minor: pypi URL uses underscores while the rest of the codebase consistently uses hyphens. |
| examples/docs_snippets/docs_snippets/integrations/hf_datasets/dataset_pipeline_example.py | End-to-end pipeline example covering load, dedup, filter, normalize, and publish stages; MaterializeResult(value=...) usage is valid; all assets are wired into Definitions. |
| examples/docs_snippets/docs_snippets_tests/test_integration_files_load.py | Adds hf_datasets/dataset_pipeline_example.py to EXCLUDED_FILES with the correct subdirectory path, preventing CI failures when dagster-hf-datasets is not installed. |
| docs/static/images/integrations/hf_datasets.svg | New SVG logo for the HF Datasets integration sidebar; referenced correctly by the docs page frontmatter. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
HFHub[(Hugging Face Hub
nyu-mll/glue QQP)] -->|hf_dataset_asset decorator| A[raw_glue_qqp
Dataset]
A -->|HFParquetIOManager| B[deduplicated_glue_qqp
Dataset]
B -->|HFParquetIOManager| C[filtered_glue_qqp
Dataset]
C -->|HFParquetIOManager| D[golden_glue_qqp
Dataset]
D --> E[publish_golden_glue
str: hub URL]
E -->|HFDatasetPublisher| HFHub2[(Hugging Face Hub
username/golden-glue-qqp)]
subgraph Resources
R1[HuggingFaceResource
cache_dir / offline]
R2[HFParquetIOManager
base_dir]
end
Reviews (6): Last reviewed commit: "update wording of doc page." | Re-trigger Greptile
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
cmpadden
left a comment
There was a problem hiding this comment.
Great - thank you @ParagEkbote !
Fixes the master break from internal build 153552 (https://buildkite.com/dagster/internal/builds/153552): - prettier/prettier errors in `dagster-oss/docs/docs/integrations/libraries/datasets/index.md` introduced by #33887 - ran `prettier --write` against the file using the repo's `.prettierrc.js` Generated with Claude Code Internal-RevId: 36151b4c3ef9a9fa21ab4684518ffd2fbe394b47
Summary & Motivation
Since the release of dagster-hf-datasets, this PR adds a documentation page and code examples describing the application and usage of the library. Could you please review the changes?
cc: @cmpadden
Test Plan
Changelog