python core dump when downloading dataset

[hansewetz]

Describe the bug

When downloading a dataset in streamed mode and exiting the program before the download completes, the python program core dumps when exiting:

terminate called without an active exception
Aborted (core dumped)

Tested with python 3.12.3, python 3.9.21

Steps to reproduce the bug

Create python venv:

python -m venv venv
./venv/bin/activate
pip install datasets==4.4.1

Execute the following program:

from datasets import load_dataset
ds = load_dataset("HuggingFaceFW/fineweb-2", 'hrv_Latn', split="test", streaming=True)
for sample in ds:
    break

Expected behavior

Clean program exit

Environment info

described above

note: the example works correctly when using datasets==3.1.0

[Lucas-Fernandes-Martins]

Hi @hansewetz I'm curious, for me it works just fine. Are you still observing the issue?

[hansewetz]

Yup ... still the same issue.
However, after adding a sleep(1) call after the for loop by accident during debugging, the program terminates properly (not a good solution though ... :-) ).
Are there some threads created that handles the download that are still running when the program exits?
Haven't had time yet to go through the code in iterable_dataset.py::IterableDataset

[Lucas-Fernandes-Martins]

Interesting, I was able to reproduce it, on a jupyter notebook the code runs just fine, as a Python script indeed it seems to never finish running (which is probably leading to the core dumped error). I'll try and take a look at the source code as well to see if I can figure it out.