Prerequisites
Feature Description
https://arxiv.org/abs/2509.02510 "Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation"
https://arxiv.org/abs/2509.23234 "p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding"
https://arxiv.org/pdf/2604.11012 "Min-k Sampling: Decoupling Truncation from Temperature Scaling via
Relative Logit Dynamics"
Motivation
Local inference lives or dies by sampler quality and setup. Although most users use temp, top-k, top-p, min-p, etc, they are simply copying and pasting what the HF model card says to do, but top-n sigma, even when ran solo, beats any sort of tweaking done to normal samplers.
Top-n sigma, even with models that have been quantized too much, and KV caches with quantization set too low, or even both at the same time, will continue to stay coherent on prompts that produce a lot of output and/or a lot of thinking, when every other sampler (or combination of samplers) cannot.
The three papers I've linked to have a chance of taking the crown from top-n sigma and may be worth implementing in llama.cpp.
Possible Implementation
No response
Prerequisites
Feature Description
https://arxiv.org/abs/2509.02510 "Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation"
https://arxiv.org/abs/2509.23234 "p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding"
https://arxiv.org/pdf/2604.11012 "Min-k Sampling: Decoupling Truncation from Temperature Scaling via
Relative Logit Dynamics"
Motivation
Local inference lives or dies by sampler quality and setup. Although most users use temp, top-k, top-p, min-p, etc, they are simply copying and pasting what the HF model card says to do, but top-n sigma, even when ran solo, beats any sort of tweaking done to normal samplers.
Top-n sigma, even with models that have been quantized too much, and KV caches with quantization set too low, or even both at the same time, will continue to stay coherent on prompts that produce a lot of output and/or a lot of thinking, when every other sampler (or combination of samplers) cannot.
The three papers I've linked to have a chance of taking the crown from top-n sigma and may be worth implementing in llama.cpp.
Possible Implementation
No response