Has anyone looked at Gigatoken-style tokenizer optimizations for llama.cpp?

[Tom-Neverwinter]

I am fairly certain we're not sleeping on this already, but just in case:

Video: https://www.youtube.com/watch?v=vFtJeZ7ifTA
Project: https://github.com/marcelroed/gigatoken

I've been following Gigatoken and was wondering whether anyone in the llama.cpp community has evaluated either the project itself or the techniques it uses.

From what I understand, Gigatoken achieves its speedups through SIMD-optimized pretokenization, cache-aware token lookup, aggressive parallelization, and minimizing overhead in the tokenization pipeline. Their benchmarks focus primarily on bulk tokenization workloads and report very large gains across tokenizers used by models we commonly run in llama.cpp (Llama, Qwen, DeepSeek, Gemma, GLM, etc.).

My assumption is that tokenization is not the primary bottleneck for most llama.cpp inference workloads. Once tokenization is complete, prompt processing/prefill and KV-cache construction are likely the dominant costs, especially for larger models and long contexts.

That said, I'm curious whether anyone has measured how much time is actually spent in tokenization versus prefill for workloads such as:

Large prompt ingestion (32K–1M contexts)
Long-context RAG
Embedding generation
Reranking
Batched inference servers
Multi-user llama-server deployments
Agent systems repeatedly tokenizing large document collections

Has anyone profiled modern llama.cpp workloads and determined what percentage of prompt-processing time is spent in tokenization?

More specifically, has there been any investigation into:

Integrating an alternative tokenizer backend
Porting similar optimization techniques into llama.cpp
Improving prompt-ingestion performance for very large contexts
Using a high-performance tokenizer as an optional component for server workloads

My suspicion is that the answer may be "tokenization is already a small fraction of total runtime, so the gains wouldn't move the needle much." If that's the case, I'd be interested to know whether there are existing benchmarks or profiling results that support that conclusion.

Mostly looking for end-to-end latency data rather than tokenizer microbenchmarks. If this has already been evaluated and ruled out, I'd love to hear the reasoning.

[yhay81]

Short answer: the idea is relevant, but the existing llama-bench prompt-processing numbers do not tell us how large the opportunity is.

At current master, llama-bench's test_prompt() creates random token IDs and passes them directly to llama_decode(); it never tokenizes text (source). So pp is intentionally a model-prefill benchmark, not an end-to-end prompt-ingestion benchmark. I could not find a current in-tree end-to-end benchmark that reports “chat template + tokenization + prefill” as separate components.

There is also real optimization headroom in the current tokenizer path. For BPE, a single call goes through unicode_regex_split, then processes each resulting word with symbol vectors and a bigram priority queue (source). I do not see document-level threading or a pretoken cache in that path. Those are exactly areas Gigatoken attacks, so it is not a case of llama.cpp already implementing all of the same techniques.

However, the headline Gigatoken numbers need careful interpretation for this use case:

• Its largest benchmark uses an 11.9 GB corpus and many cores; that measures sustained bulk throughput, not single-request latency.
• The fastest API reads files directly and can split work for maximum parallelism. The Hugging Face compatibility API deliberately gives up some of that advantage.
• llama.cpp loads tokenizer vocabulary/merges/normalization and special-token behavior from GGUF. A backend must produce byte-for-byte identical token IDs for all of those cases—not just common Llama/Qwen prose—before it can replace llama_tokenize.
• Embedding/reranking/bulk-ingestion servers are much more likely to benefit than an interactive request whose model prefill is already much longer than tokenization.

A useful way to measure this today is:

1. Start one warm llama-server with prompt caching disabled for the test.
2. Apply the real chat template first (POST /apply-template), because template work is otherwise easily mixed into “tokenization.”
3. Measure the same resulting string through the documented POST /tokenize endpoint, with with_pieces: false. HTTP and JSON overhead make this an upper bound; an in-process timer around llama_tokenize is the accurate measurement.
4. Measure prefill separately with the same token IDs. Do not use llama-bench pp as the tokenizer measurement—it starts after this step.
5. Repeat for 32K, 128K, 512K and 1M-token inputs, warm and cold caches, and at concurrency 1/N. Report p50/p95 latency, input MB/s, tokens/s and CPU usage.
6. Validate every candidate backend's token array against llama.cpp across BPE, SentencePiece, byte fallback, Unicode edge cases, added/special tokens, and chat templates before comparing speed.

The simple break-even relation is useful: for N tokens, b input bytes/token, tokenizer throughput S bytes/s and prefill throughput P tokens/s, the tokenizer fraction is approximately

(bN/S) / (bN/S + N/P) = bP / (S + bP).

That shows why prompt length alone does not make tokenization dominant: both costs grow with N (while attention details can make prefill grow worse). What matters is how fast the model backend is relative to the tokenizer. A very fast GPU serving a small embedding/reranker model, or a CPU farm repeatedly ingesting documents, can cross that threshold; a large generative model usually will not.

My practical conclusion would be:

• Interactive generation: likely a small end-to-end win; measure before adding a dependency.
• Embeddings, reranking, corpus ingestion, high-concurrency servers: plausible and worth a dedicated benchmark.
• Repeated agent/RAG documents: caching already-tokenized immutable chunks may give a larger, simpler win than retokenizing them faster.
• Implementation direction: first add an end-to-end/tokenizer benchmark and correctness corpus; then evaluate porting the SIMD/pretoken-cache techniques. Embedding Gigatoken itself is possible license-wise (MIT), but its Rust/Python-oriented integration and portability cost make a native technique port or optional backend a separate design decision.

So I would not rule this out as “tokenization is obviously negligible.” The current benchmark suite simply excludes the stage needed to prove that claim.