Under some circumstances joblib's pre-fetching from the batch_generator inside SemiValueValuation.fit() consumes way more samples than instructed, even after passing batch_size=1 and pre_dispatch="n_jobs" in the call to Parallel. This renders the mechanism that tries to interrupt the processing by interrupting the sampler via skip_indices useless.
A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use MinUpdates(10) as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.
Under some circumstances joblib's pre-fetching from the batch_generator inside
SemiValueValuation.fit()consumes way more samples than instructed, even after passingbatch_size=1andpre_dispatch="n_jobs"in the call toParallel. This renders the mechanism that tries to interrupt the processing by interrupting the sampler viaskip_indicesuseless.A good way to test this is to use a powerset sampler with a sequential index iteration which generates, say 1000 samples per index. Then use
MinUpdates(10)as stopping criterion. Even though skip_indices will be correctly updated to skip the first index after a few updates, the processing will continue for a long while before moving on to the next index.