Add test to verify integrity of loaded datasets, Avoid copying=True for LoadHF and two more loaders. Remove loader cache (python knows how to manage main memory).

[dafnapension]

avoid copying=true (upon iterating over the loaded dataset) for the streams generated by the loader for loaders which watch out after the datasets they load - letting no one modify them. This is the case for LoadHF, LoadCSV, and LoadSKLearn. For the rest -- copying=True remains. Also, remove the loader cache. We are not sure what goes there (a whole listed dataset? just its generators?) and python better manages its main memory knowing the whole picture (and remove to file system whole pages as it sees fit)

E.g that demonstrates how the HF dataset, returned by LoadHF, is not modifiable.

    from datasets import load_dataset as hf_load_dataset
    ds = hf_load_dataset("PrimeQA/clapnq_passages")
    print(ds)
    iter_train = iter(ds["train"])
    iter_train2 = iter(ds["train"])
    for _ in range(5):
        instance = next(iter_train)
        print(instance["id"])
        instance["id"] = 5
    
    for _ in range(5):
        instance = next(iter_train2)
        print(instance["id"])
        
    iter_train = iter(ds["train"])
    iter_train2 = iter(ds["train"])
    for _ in range(5):
        instance = next(iter_train)
        old_value = instance.pop("id")
        instance["id_new"] = old_value
    
    for _ in range(5):
        instance = next(iter_train2)
        print(instance["id"])
        print("id_new" in instance)

    DatasetDict({
        train: Dataset({
            features: ['id', 'text', 'title'],
            num_rows: 178890
        })
    })
    827849752_115-357
    827849752_358-743
    827849752_744-1186
    827849752_1187-1426
    827849752_1604-3189
    827849752_115-357
    827849752_358-743
    827849752_744-1186
    827849752_1187-1426
    827849752_1604-3189
    827849752_115-357
    False
    827849752_358-743
    False
    827849752_744-1186
    False
    827849752_1187-1426
    False
    827849752_1604-3189
    False

[elronbandel]

This is unnecassry as inside the log limited loading there is a mechanism to log only once. Which means it is logging only once and only when the data is actually loaded.

[dafnapension]

I think that the mechanism hides problem. Like it hid from you the loop in LoadCSV.
This logging is cleaner, and happens just once, and does not introduce new variables.

[elronbandel]

In the case of lazy loader we never reach that point, so for most of the loaders the copying=True is not affecting much, no?

[dafnapension]

Currently, we only reach this line. No Loader returns a MultiStream from its load_iterables.

[dafnapension]

closed for no public interest