Constructive criticism for Sentence Transformers.

[BBC-Esq]

I give the Sentence Transformers and Tokenizers libraries some constructive criticism in my new article, but like any good journalist, welcome any criticism if I misunderstood any part of the Sentence Transformers library...just let me know...I'm always open to amending my findings...

https://medium.com/@vici0549/sentence-transformers-and-the-huggingface-tokenizer-are-broken-at-scale-heres-what-i-found-after-acd03d85c919

[tomaarsen]

Hello!

Thanks for sending over the detailed post! I'll mostly cover the Sentence Transformers portion here, and leave the tokenizers part for huggingface/tokenizers#1954.

Sentence Transformers indeed defaults to a batch size of 32. I understand that this is rather high for modern and large embedding models, especially with large sequence lengths, but it's actually much too small for small models. There's simply not a single default that works with all situations. For example:

"""Find the optimal batch_size for encoding by benchmarking several values."""

import time
from datasets import load_dataset
from sentence_transformers import SentenceTransformer

# Load model
model = SentenceTransformer("all-mpnet-base-v2", model_kwargs={"torch_dtype": "bfloat16"})

# Load a subset of Natural Questions answers
dataset = load_dataset("sentence-transformers/natural-questions", split="train[:10000]")
sentences = dataset["answer"]

# Benchmark different batch sizes
batch_sizes = [8, 16, 32, 64, 128, 256, 512]
results = {}

for bs in batch_sizes:
    start = time.perf_counter()
    model.encode(sentences, batch_size=bs, show_progress_bar=False)
    elapsed = time.perf_counter() - start
    throughput = len(sentences) / elapsed
    results[bs] = throughput
    print(f"batch_size={bs:>4d}  |  {elapsed:.2f}s  |  {throughput:.0f} sentences/sec")

best_bs = max(results, key=results.get)
print(f"\nBest batch_size: {best_bs} ({results[best_bs]:.0f} sentences/sec)")

batch_size=   8  |  28.18s  |  355 sentences/sec
batch_size=  16  |  17.75s  |  563 sentences/sec
batch_size=  32  |  15.08s  |  663 sentences/sec
batch_size=  64  |  12.82s  |  780 sentences/sec
batch_size= 128  |  11.79s  |  848 sentences/sec
batch_size= 256  |  11.42s  |  876 sentences/sec
batch_size= 512  |  11.76s  |  851 sentences/sec

Best batch_size: 256 (876 sentences/sec)

I think a solid solution is to be more explicit with recommending adjusting the batch size in my documentation. I've opened a PR here: #3672. Feel free to let me know what you think!

Beyond that, I've been talking with the transformers team whether Sentence Transformers can take advantage of some new functionality from transformers regarding padding-free inference using flash-attention. I'll try to look into this in the coming days, as it could certainly speed up inference overall if the model architecture & user environment support it.

One small comment regarding the tokenizer: I've never encountered this before on Windows myself, but I wrote a script to try and reproduce it:

from datasets import load_dataset
from transformers import AutoTokenizer
from tqdm import trange

tokenizer_batch_size = 32

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Embedding-0.6B")
dataset = load_dataset("miriad/miriad-5.8M", split="train")
passages = dataset["passage_text"]  # 5.82M passages of up to 10k characters

for batch_start in trange(0, len(passages), tokenizer_batch_size, desc="Tokenizing long passages"):
    batch_passages = passages[batch_start : batch_start + tokenizer_batch_size]
    encodings = tokenizer(batch_passages, truncation=True, padding=True, return_tensors="pt")

It's currently 81% completed:

Tokenizing long passages:  81%|██████████████████████████████████████████████████████████████████████████████████████████████▎                      | 146650/181936 [1:38:44<23:16, 25.2it/s]

Which means it has successfully tokenized 4.692m texts from https://huggingface.co/datasets/miriad/miriad-5.8M/viewer/default/train, from the passage_text column, which is about ~4k characters on average (up to 10k). Could you perhaps check whether this script fails for you? We'd really like to reproduce this, so we can try and prevent it from happening again.

Edit: It completed:

Tokenizing long passages: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 181936/181936 [1:59:08<00:00, 25.45it/s]

• Tom Aarsen

[BBC-Esq]

Yes, since I'll try what you suggest and report back in a few days hopefully once I get some other things off my plate. I also responded to your post in the Tokenizers forum...thanks so much BTW.

[BBC-Esq]

Just wanted to let you know I haven't forgotten...work and RL stuff and other projects have been hectic but I do plan to revisit this asap!