[Bug]: DefaultHuggingfaceDataLoadStrategy passes unsupported limit to datasets.load_dataset #1091

Before Reporting

• I have asked the Data-Juicer Q&A Copilot (available on Doc Site, DingTalk, or Discord), but the problem still persists.
• I have pulled the latest code of main branch to run again and the bug still existed.
• I have read the README carefully and no error occurred during the installation process. (Otherwise, we recommend that you can ask a question using the Question template)

Search before reporting

• I have searched the Data-Juicer issues and found no similar bugs.

OS

macOS 15 (the code path is platform independent)

Installation Method

from source

Data-Juicer Version

main @ c31683c

Python Version

3.11

Describe the bug

DefaultHuggingfaceDataLoadStrategy.load_data passes limit= to datasets.load_dataset (load_strategy.py#L376), and limit is also listed in optional_fields (#L363). datasets.load_dataset has no limit parameter, so unknown kwargs are forwarded to the builder config and the call raises, even when limit is None. As far as I can tell this affects every config using type: remote, source: huggingface on the default executor. limit is advertised as an optional field but cannot work as a row cap.

I did not run this against the Hub (no network in my environment); I reproduced it with the packaged json builder, which goes through the same load_dataset kwargs handling.

To Reproduce

1. Use a config with dataset: {configs: [{type: remote, source: huggingface, path: <any dataset>}]} on the default executor.
2. Minimal equivalent of the call the strategy makes:

import datasets
datasets.load_dataset("json", data_files="t.jsonl", split=None, limit=None, name=None, data_dir=None, num_proc=1)

Output with datasets 5.1.0: TypeError: JsonConfig.__init__() got an unexpected keyword argument 'limit'

Note: data_juicer itself was not importable in my repro environment, so this exercises the datasets call directly rather than the full strategy. tests/core/data/test_load_strategy.py mocks load_dataset, so it would not catch this.

Configs

dataset:
  configs:
    - type: remote
      source: huggingface
      path: <any dataset>

Logs

TypeError: JsonConfig.__init__() got an unexpected keyword argument 'limit'

Screenshots

No response

Expected Behavior

The dataset loads. If limit is set, only that many rows are kept.

Additional Context

Suggested fix: do not pass limit to load_dataset; apply it after loading (ds.select(range(min(limit, len(ds))))) or via split=f"{split}[:{limit}]".

Separate observation, not part of the bug above: when split is not set, load_dataset returns a DatasetDict, which unify_format may not handle as a single dataset. Defaulting split to "train" might be worth considering.