Unique indices required?

[gklambauer]

Great work, thanks! What if I have a weighted sampling scheme, where some samples are drawn with higher probabilities. I tried to provide "indices=idxSamples" to the arrayDataSource, in which "idxSamples can contain an integer index multiple times. This seems to lead to problems, so my question is: do you assume that the indices provided to the argument "indices" are unique?

[Britefury]

Hi,
I apologise for the delayed reply.
I just tried the following example:

x = np.arange(10)*10
ndx = np.concatenate([np.arange(i) for i in range(1,11)], axis=0)
print('Data={}'.format(x))
print('Indices={}'.format(ndx))
hist_of_indices = np.bincount(ndx, minlength=10)
print('Histogram of indices={}'.format(hist_of_indices))

ads = data_source.ArrayDataSource([x], indices=ndx)

p = []
for (bx,) in ads.batch_iterator(batch_size=10, shuffle=True):
    p.append(bx)
p = np.concatenate(p, axis=0)
sample_hist = np.bincount(p, minlength=100)
print('Samples drawn={}'.format(p))
print('Histogram of samples drawn={}'.format(sample_hist))

Produces the following output:

Data=[ 0 10 20 30 40 50 60 70 80 90]
Indices=[0 0 1 0 1 2 0 1 2 3 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 0
 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 9]
Histogram of indices=[10  9  8  7  6  5  4  3  2  1]
Samples drawn=[60 50 10 30  0 80 60 20 50 20 40 20 20 30 20 30 70 50 50 20  0 40 60 40 30
 10 30 50 40 70 70 10  0 80 60  0  0 40 10 90  0 10 30 10 40 20 10 10 30  0
  0 20  0  0 10]
Histogram of samples drawn=[10  0  0  0  0  0  0  0  0  0  9  0  0  0  0  0  0  0  0  0  8  0  0  0  0
  0  0  0  0  0  7  0  0  0  0  0  0  0  0  0  6  0  0  0  0  0  0  0  0  0
  5  0  0  0  0  0  0  0  0  0  4  0  0  0  0  0  0  0  0  0  3  0  0  0  0
  0  0  0  0  0  2  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0]

It seemed to work fine. What problems are you encountering specifically?

Further more, I think I need to implement some sort of PyTorch style sampler system to allow more configurable sample iteration options e.g. sample weighting.

[gklambauer]

I found out that the errors rather arise from the last (typically incomplete) batch. Sorry bringing this up! Thanks for BatchUp -- great stuff!!!

[Britefury]

I tried changing the batch size in the above example to 7 or 9 and found that it still worked. Are you still encountering problems? If so can you provide any more detail?

Thank you; glad you like BatchUp!