Improve API to use a custom batch_sampler in a SentenceTransformerTrainer

[alonme]

Currently only specific batch_sampler values are possible
There seems to be a need to enable users to create custom batch samplers

Examples from issues:

1. Batch sampler #3123
2. Using DataLoader in v3? #2707
3. MNRL with Multiple hard negatives per query and NoDuplicatesBatchSampler #2954

I believe that the current solution (that i found the hard way) that is suggested in the issues is to subclass SentenceTransformerTrainer and override get_batch_sampler, is not documented well enough and isn't straightforward

Is there any reason not to just accept anything that inherits from DefaultBatchSampler (for example) as a parameter?

[tomaarsen]

Hello!

The current implementation is just an extension of transformers TrainingArguments, where e.g. optimizer options are also provided by passing a string to optim with one of these strings: https://github.com/huggingface/transformers/blob/12ba96aa3cb3e4ed2a3ffb77b59f53f8ce9ac1fa/src/transformers/training_args.py#L143-L185

Having said that, the transformers Trainer class does accept an optimizers instance where you can pass initialized instances.

I think I would be okay with updating the API such that a subclass of DefaultBatchSampler can be used. We'd first have to extend DefaultBatchSampler so it also accepts valid_label_columns and generator (but keeps them unused), and then people who subclass it can use those still.
I think I can have a look at this soon.

• Tom Aarsen

[alonme]

@tomaarsen Hey! Thanks for the reply,

I Assume i can also try and tackle that, would that be ok?

[tomaarsen]

Definitely!
If possible, perhaps the updated DefaultBatchSampler can be the superclass of the other batch samplers as well - that would make it more obvious that it should be subclassed.
If you're interested, we can also apply this approach for the multi-dataset batch samplers.

• Tom Aarsen