Tuning with large datasets

[linfeng-du]

Describe the issue:
When tuning with large datasets on a single machine with multiple gpus. Is there an easy way to just load the dataset for once and then share it across different trials? (Loading it multiple times in parallel may cause OOM)

[acured]

Hi @CapFreddy , we have a plan about this, but for now, there is no way to do that from NNI.
Or you can pre-download the data, and modify trial code load data from local.

[conceptofmind]

Hi @acured ,

I am currently having a similar issue regarding using a 4 GB dataset in a parquet file.

I receive the errors:

PayloadTooLarge: Pickle too large when trying to dump <torch.utils.data.dataset.Subset object at 0x7f64350625e0>. This might be caused by classes that are not decorated by @nni.trace. Another option is to force bytes pickling and try to raise pickle_size_limit.

ValueError: Serialization failed when trying to dump the model because payload too large (larger than 64 KB). This is usually caused by pickling large objects (like datasets) by mistake. See the full error traceback for details and https://nni.readthedocs.io/en/stable/NAS/Serialization.html for how to resolve such issue.

As well as:
3: 0xcfe71e v8::Utils::ReportOOMFailure
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Any advice would be great!

Thank you,

Eric

[mashu]

I am having similar issue,

  File "/home/mateusz/miniconda3/envs/pytorch/lib/python3.10/site-packages/nni/common/serializer.py", line 864, in _json_tricks_any_object_encode
    raise PayloadTooLarge(f'Pickle too large when trying to dump {obj}. This might be caused by classes that are '
nni.common.serializer.PayloadTooLarge: Pickle too large when trying to dump <__main__.MiAIRRDataset object at 0x7fbb47c9c7f0>. This might be caused by classes that are not decorated by @nni.trace. Another option is to force bytes pickling and try to raise pickle_size_limit.
[2022-12-11 08:36:44] Stopping experiment, please wait...
[2022-12-11 08:36:44] Dispatcher exiting...

Basically the only way is to increase pickle limit size, but than everything else gets broken. After view trails GUI crashes, also logs become gigabytes..

-rw-r----- 1 mateusz mateusz 1.6G Dec 11 08:21 nnimanager.log

It's basically broken with bigger and non-serializable data, and I need complete data.