Skip to content

[LLM:Bugfix] Fix WikiText calibration dataset loading#4566

Merged
wangzhaode merged 1 commit into
alibaba:masterfrom
LudovicoYIN:llm-fix-wikitext-dataset-loading
Jun 25, 2026
Merged

[LLM:Bugfix] Fix WikiText calibration dataset loading#4566
wangzhaode merged 1 commit into
alibaba:masterfrom
LudovicoYIN:llm-fix-wikitext-dataset-loading

Conversation

@LudovicoYIN

Copy link
Copy Markdown
Contributor

Description

Fix LLM calibration dataset loading with recent datasets / huggingface_hub versions.

Recent HuggingFace Hub clients require dataset repo IDs to include a namespace. The old load_dataset("wikitext", ...) path now fails with an invalid HF URI error. This updates built-in WikiText calibration loading to Salesforce/wikitext.

Also updates download_data.py to accept dataset specs like Salesforce/wikitext/wikitext-2-raw-v1 by treating the final path component as the dataset config and the preceding components as the repo ID.

Module

LLM

Type

  • Feature
  • Bugfix
  • Perf
  • Refact
  • Style
  • Doc
  • Test
  • Chore

Checklist

  • Commit message follows [Module:Type] Description format
  • Code compiles without errors
  • Tested on relevant platform(s)
  • No unrelated format or style changes included

@wangzhaode wangzhaode left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! The root cause is confirmed — recent HuggingFace Hub versions require a namespaced repo ID, and the old load_dataset('wikitext', ...) call fails with HfUriError. The updated Salesforce/wikitext identifier and the new three-part spec parsing are both correct.

@LudovicoYIN

Copy link
Copy Markdown
Contributor Author

Thanks your review.

@wangzhaode wangzhaode self-assigned this Jun 25, 2026
@wangzhaode
wangzhaode merged commit 3533cd0 into alibaba:master Jun 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants