no main-memory cache to loaders

[dafnapension]

and no copying=true for the streams generated by the loader for loaders which watch out after the datasets they load - letting no one modify them. This is the case for LoadHF, LoadCSV, and LoadSKLearn. For the rest -- copying=True remains.

E.g that demonstrates how the HF dataset, returned by LoadHF, is not modifiable.

    from datasets import load_dataset as hf_load_dataset
    ds = hf_load_dataset("PrimeQA/clapnq_passages")
    print(ds)
    iter_train = iter(ds["train"])
    iter_train2 = iter(ds["train"])
    for _ in range(5):
        instance = next(iter_train)
        print(instance["id"])
        instance["id"] = 5
    
    for _ in range(5):
        instance = next(iter_train2)
        print(instance["id"])
        
    iter_train = iter(ds["train"])
    iter_train2 = iter(ds["train"])
    for _ in range(5):
        instance = next(iter_train)
        old_value = instance.pop("id")
        instance["id_new"] = old_value
    
    for _ in range(5):
        instance = next(iter_train2)
        print(instance["id"])
        print("id_new" in instance)

    DatasetDict({
        train: Dataset({
            features: ['id', 'text', 'title'],
            num_rows: 178890
        })
    })
    827849752_115-357
    827849752_358-743
    827849752_744-1186
    827849752_1187-1426
    827849752_1604-3189
    827849752_115-357
    827849752_358-743
    827849752_744-1186
    827849752_1187-1426
    827849752_1604-3189
    827849752_115-357
    False
    827849752_358-743
    False
    827849752_744-1186
    False
    827849752_1187-1426
    False
    827849752_1604-3189
    False

[elronbandel]

This is unnecassry as inside the log limited loading there is a mechanism to log only once. Which means it is logging only once and only when the data is actually loaded.

[dafnapension]

I think that the mechanism hides problem. Like it hid from you the loop in LoadCSV.
This logging is cleaner, and happens just once, and does not introduce new variables.

[elronbandel]

In the case of lazy loader we never reach that point, so for most of the loaders the copying=True is not affecting much, no?

[dafnapension]

Currently, we only reach this line. No Loader returns a MultiStream from its load_iterables.