Feature Request: Implement Power Law sampling

[Geechan]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I would like to request the implementation of the new Power Law sampler that has recently gotten an outstanding PR implementation in llama.cpp. Originally authored and created by @MrJackSpade here, and a working implementation in upstream llama.cpp ported and refined by @ddh0 here.

What is it?

This sampler effectively targets the 'creative yet coherent' token options in the middle of the sampling range, allowing for the end user to specify a target value that the model will swing towards. This changes the probability distribution in a less destructive way compared to adjusting temperature, and allows for more creativity while breaking a lot of repetitive model patterns and phrases (or, as we know it, slop). This can be considered an evolved version of Microstat and XTC, with similar goals, but with less reliance on RNG or complexity, setting an adaptive target instead of a static one determined by chance probability.

Adaptive target tracking

The sampler maintains a weighted history of the original probabilities of selected tokens. If recent selections have been higher-probability than the target, it compensates by temporarily lowering the effective target, and vice-versa. This keeps the average selection probability near your configured target over time.

Tunable Parameters

Flag	Description	Valid range	Default
--power-law-target	Select tokens near this probability. Negative = disabled.	[0.0, 1.0]	-1.0
--power-law-decay	Decay rate for target adaptation over time. Effective history length ≈ 1/(1-decay) tokens.	[0.0, 0.99]	0.9

Negative target values should disable the sampler and just sample a token from the un-transformed distribution. By setting the default target to -1.0, the sampler will be disabled by default. This is probably preferable, since it's a specialized sampler.

Usage notes

This sampler must be last in the chain, like the existing greedy, dist, or mirostat samplers, because it selects a token ID rather than just transforming logits. As a result, an implementation that guarantees Power Law to be at the end of the chain is probably preferable; see here.

Motivation

This sampler effectively functions as a smart, dynamic XTC, functioning very well for creative tasks of all descriptions. It can also additionally be used for non-creative tasks quite effectively by setting a higher target value (0.8-0.9), leaning closer towards the original probabilities.

By implementing it in ik_llama, it would add another powerful sampler option for the end user. The actual sampler logic is quite simple and should be easy to port over.

Possible Implementation

A reference .gist implementation can be found here, written for llama.cpp, which contains all the sampler logic:

https://gist.github.com/ddh0/3870fba61e482e0ad27f1812e32581bf

The outstanding PR request for upstream llama.cpp, with a detailed, technical explanation on the sampler and the parameters chosen to be visible to the end user, alongside other pertinent information:

ggml-org/llama.cpp#17927

[magikRUKKOLA]

Any PPL info?..

[EDIT]: wait a sec. the llama-bench doesn't use any samplers, right? So is there any way to test the performance of the method described?

[Geechan]

Any PPL info?..

[EDIT]: wait a sec. the llama-bench doesn't use any samplers, right? So is there any way to test the performance of the method described?

I suggest trying the sampler out for yourself in the llama.cpp branch posted above if you want to see the effects in action. AFAIK there's no reliable way to measure the PPL of this, as with all samplers. The effects are very noticeable in creative writing without affecting coherency due to the self correcting algorithm preventing the sampler from straying far from the target. With such applications, you want an element of randomness and unpredictability to keep things engaging, and PPL information wouldn't tell the whole story (if any), anyway.

[ikawrakow]

I saw the PR in mainline. Wonder why it has not been merged 8 days after submission?

[Ph0rk0z]

They're probably not too receptive to creative stuff anymore. I saw that PR too and thought "man, IK is never gonna want to add that, probably won't get to try it". I'd also have to manually slip it into sillytavern to really test it and demotivating to do it just for a PR.

[ikawrakow]

@Ph0rk0z

They're probably not too receptive to creative stuff anymore.

Is it that? Or is it just the underwhelming level of excitement (I count 3 people being excited about this sampler in the mainline PR). You can generate some extra excitement and motivation to implement in ik_llama.cpp by actually trying it and being impressed.

[ddh0]

I saw the PR in mainline. Wonder why it has not been merged 8 days after submission?

They are waiting on a PR to implement backend sampling, then they will merge power law

ggml-org/llama.cpp#17004

[Geechan]

They're probably not too receptive to creative stuff anymore. I saw that PR too and thought "man, IK is never gonna want to add that, probably won't get to try it". I'd also have to manually slip it into sillytavern to really test it and demotivating to do it just for a PR.

Just as a heads up: you can actually use the sampler right now in SillyTavern under chat completions. Start up llama.cpp with the PR branch, connect to chat completions (add /v1 at the end of your llama.cpp URL), enable chat completions, and modify the Additional Parameters section as shown: