fix(deps): update dependency datasets to v3.6.0

[konflux-internal-p02]

This PR contains the following updates:

Package	Change	Age	Confidence
datasets	==3.1.0 → ==3.6.0

---

Release Notes

huggingface/datasets (datasets)

v3.6.0

Compare Source

Dataset Features

• Enable xet in push to hub by @​lhoestq in #​7552
  • Faster downloads/uploads with Xet storage
  • more info: #​7526

Other improvements and bug fixes

• Add try_original_type to DatasetDict.map by @​yoshitomo-matsubara in #​7544
• Avoid global umask for setting file mode. by @​ryan-clancy in #​7547
• Rebatch arrow iterables before formatted iterable by @​lhoestq in #​7553
• Document the HF_DATASETS_CACHE environment variable in the datasets cache documentation by @​Harry-Yang0518 in #​7532
• fix regression by @​lhoestq in #​7558
• fix: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames (#​7517) by @​giraffacarp in #​7521
• Remove aiohttp from direct dependencies by @​akx in #​7294

New Contributors

• @​ryan-clancy made their first contribution in #​7547
• @​Harry-Yang0518 made their first contribution in #​7532
• @​giraffacarp made their first contribution in #​7521
• @​akx made their first contribution in #​7294

Full Changelog: huggingface/datasets@3.5.1...3.6.0

v3.5.1

Compare Source

Bug fixes

• support pyarrow 20 by @​lhoestq in #​7540
  • Fix pyarrow error TypeError: ArrayExtensionArray.to_pylist() got an unexpected keyword argument 'maps_as_pydicts'
• Write pdf in map by @​lhoestq in #​7487

Other improvements

• update fsspec 2025.3.0 by @​peteski22 in #​7478
• Support underscore int read instruction by @​lhoestq in #​7488
• Support skip_trying_type by @​yoshitomo-matsubara in #​7483
• pdf docs fixes by @​lhoestq in #​7519
• Remove conditions for Python < 3.9 by @​cyyever in #​7474
• mention av in video docs by @​lhoestq in #​7523
• correct use with polars example by @​SiQube in #​7524
• chore: fix typos by @​afuetterer in #​7436

New Contributors

• @​peteski22 made their first contribution in #​7478
• @​yoshitomo-matsubara made their first contribution in #​7483
• @​SiQube made their first contribution in #​7524
• @​afuetterer made their first contribution in #​7436

Full Changelog: huggingface/datasets@3.5.0...3.5.1

v3.5.0

Compare Source

Datasets Features

• Introduce PDF support (#​7318) by @​yabramuvdi in #​7325

>>> from datasets import load_dataset, Pdf
>>> repo = "path/to/pdf/folder"  # or username/dataset_name on Hugging Face
>>> dataset = load_dataset(repo, split="train")
>>> dataset[0]["pdf"]
<pdfplumber.pdf.PDF at 0x1075bc320>
>>> dataset[0]["pdf"].pages[0].extract_text()
...

What's Changed

• Fix local pdf loading by @​lhoestq in #​7466
• Minor fix for metadata files in extension counter by @​lhoestq in #​7464
• Priotitize json by @​lhoestq in #​7476

New Contributors

• @​yabramuvdi made their first contribution in #​7325

Full Changelog: huggingface/datasets@3.4.1...3.5.0

v3.4.1

Compare Source

Bug Fixes

• Fix data_files filtering by @​lhoestq in #​7459

Full Changelog: huggingface/datasets@3.4.0...3.4.1

v3.4.0

Compare Source

Dataset Features

• Faster folder based builder + parquet support + allow repeated media + use torchvideo by @​lhoestq in #​7424

  • /!\ Breaking change: we replaced decord with torchvision to read videos, since decord is not maintained anymore and isn't available for recent python versions, see the video dataset loading documentation here for more details. The Video type is still marked as experimental is this version

  from datasets import load_dataset, Video
  
  dataset = load_dataset("path/to/video/folder", split="train")
  dataset[0]["video"]  # <torchvision.io.video_reader.VideoReader at 0x1652284c0>

  • faster streaming for image/audio/video folder from Hugging Face
  • support for metadata.parquet in addition to metadata.csv or metadata.jsonl for the metadata of the image/audio/video files

• Add IterableDataset.decode with multithreading by @​lhoestq in #​7450

  • even faster streaming for image/audio/video folder from Hugging Face if you enable multithreading to decode image/audio/video data:

  dataset = dataset.decode(num_threads=num_threads)

• Add with_split to DatasetDict.map by @​jp1924 in #​7368

General improvements and bug fixes

• fix: None default with bool type on load creates typing error by @​stephantul in #​7426
• Use pyupgrade --py39-plus by @​cyyever in #​7428
• Refactor string_to_dict to return None if there is no match instead of raising ValueError by @​ringohoffman in #​7435
• Fix small bugs with async map by @​lhoestq in #​7445
• Fix resuming after ds.set_epoch(new_epoch) by @​lhoestq in #​7451
• minor docs changes by @​lhoestq in #​7452

New Contributors

• @​stephantul made their first contribution in #​7426
• @​cyyever made their first contribution in #​7428
• @​jp1924 made their first contribution in #​7368

Full Changelog: huggingface/datasets@3.3.2...3.4.0

v3.3.2

Compare Source

Bug fixes

• Attempt to fix multiprocessing hang by closing and joining the pool before termination by @​dakinggg in #​7411
• Gracefully cancel async tasks by @​lhoestq in #​7414

Other general improvements

• Update use_with_pandas.mdx: to_pandas() correction in last section by @​ibarrien in #​7407
• Fix a typo in arrow_dataset.py by @​jingedawang in #​7402

New Contributors

• @​dakinggg made their first contribution in #​7411
• @​ibarrien made their first contribution in #​7407
• @​jingedawang made their first contribution in #​7402

Full Changelog: huggingface/datasets@3.3.1...3.3.2

v3.3.1

Compare Source

Bug fixes

• Fix filter speed regression by @​lhoestq in #​7408

Full Changelog: huggingface/datasets@3.3.0...3.3.1

v3.3.0

Compare Source

Dataset Features

• Support async functions in map() by @​lhoestq in #​7384

  • Especially useful to download content like images or call inference APIs

  prompt = "Answer the following question: {question}. You should think step by step."
  async def ask_llm(example):
      return await query_model(prompt.format(question=example["question"]))
  ds = ds.map(ask_llm)

• Add repeat method to datasets by @​alex-hh in #​7198

  ds = ds.repeat(10)

• Support faster processing using pandas or polars functions in IterableDataset.map() by @​lhoestq in #​7370

  • Add support for "pandas" and "polars" formats in IterableDatasets
  • This enables optimized data processing using pandas or polars functions with zero-copy, e.g.

  ds = load_dataset("ServiceNow-AI/R1-Distill-SFT", "v0", split="train", streaming=True)
  ds = ds.with_format("polars")
  expr = pl.col("solution").str.extract("boxed\\{(.*)\\}").alias("value_solution")
  ds = ds.map(lambda df: df.with_columns(expr), batched=True)

• Apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets by @​alex-hh in #​7207

  • IterableDatasets with "numpy" format are now much faster

What's Changed

• don't import soundfile in tests by @​lhoestq in #​7340
• minor video docs on how to install by @​lhoestq in #​7341
• Fix typo in arrow_dataset by @​AndreaFrancis in #​7328
• remove filecheck to enable symlinks by @​fschlatt in #​7133
• Webdataset special columns in last position by @​lhoestq in #​7349
• Bump hfh to 0.24 to fix ci by @​lhoestq in #​7350
• fsspec 2024.12.0 by @​lhoestq in #​7352
• changes to MappedExamplesIterable to resolve #​7345 by @​vttrifonov in #​7353
• Catch OSError for arrow by @​lhoestq in #​7348
• Remove .h5 from imagefolder extensions by @​lhoestq in #​7374
• Add Pandas, PyArrow and Polars docs by @​lhoestq in #​7382
• Optimized sequence encoding for scalars by @​lukasgd in #​7393
• Update docs by @​lhoestq in #​7395
• Update README.md by @​lhoestq in #​7396
• Release: 3.3.0 by @​lhoestq in #​7398

New Contributors

• @​AndreaFrancis made their first contribution in #​7328
• @​vttrifonov made their first contribution in #​7353
• @​lukasgd made their first contribution in #​7393

Full Changelog: huggingface/datasets@3.2.0...3.3.0

v3.2.0

Compare Source

Dataset Features

• Faster parquet streaming + filters with predicate pushdown by @​lhoestq in #​7309
  • Up to +100% streaming speed
  • Fast filtering via predicate pushdown (skip files/row groups based on predicate instead of downloading the full data), e.g.

    from datasets import load_dataset
    filters = [('date', '>=', '2023')]
    ds = load_dataset("HuggingFaceFW/fineweb-2", "fra_Latn", streaming=True, filters=filters)

Other improvements and bug fixes

• fix conda release worlflow by @​lhoestq in #​7272
• Add link to video dataset by @​NielsRogge in #​7277
• Raise error for incorrect JSON serialization by @​varadhbhatnagar in #​7273
• support for custom feature encoding/decoding by @​alex-hh in #​7284
• update load_dataset doctring by @​lhoestq in #​7301
• Let server decide default repo visibility by @​Wauplin in #​7302
• fix: update elasticsearch version by @​ruidazeng in #​7300
• Fix typing in iterable_dataset.py by @​lhoestq in #​7304
• Updated inconsistent output in documentation examples for ClassLabel by @​sergiopaniego in #​7293
• More docs to from_dict to mention that the result lives in RAM by @​lhoestq in #​7316
• Release: 3.2.0 by @​lhoestq in #​7317

New Contributors

• @​ruidazeng made their first contribution in #​7300
• @​sergiopaniego made their first contribution in #​7293

Full Changelog: huggingface/datasets@3.1.0...3.2.0

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

---

• If you want to rebase/retry this PR, check this box

---

To execute skipped test pipelines write comment /ok-to-test.

---

Documentation

Find out how to configure dependency updates in MintMaker documentation or see all available configuration options in Renovate documentation.