How to deal with large amounts of data?

[fabioperez]

I am trying to use Xcessiv to an image classification project (together with Keras or PyTorch). By reading the walkthrough, I found that I have to pass the entire dataset as a (X, y) tuple. This is unfeasible for a large image dataset. How can I outcome this problem?

One solution that I thought about was to pass image paths as the X, and let fit load the data lazily. Is this the best approach?

PS: Thanks for creating and maintaining Xcessiv!

[reiinakano]

Hi @fabioperez, unfortunately I built Xcessiv without deep learning/image processing in mind. It is certainly not feasible to use Xcessiv with the large amounts of data necessary for deep learning (which are usually passed in to a neural network as batches, anyway).

Also, you give me too much credit. While I built Xcessiv, I certainly haven't had the time to maintain it in the last year, and I don't see that changing soon. :(