Is the lexer fuel user refillable?

[afg1]

I've been moving some of my code over to the 0.2.0 version of guidance, and I've hit a problem which I don't see in the 0.1.16 version. Namely, the lexer is refusing to do things in the new version which the old one was happy with.

After some creative debugging, I found that the rust lexer is running out of 'fuel' when attempting to select a substring from a paragraph (for example).

I think I understand that the fuel idea is to control how long the longest running lexer is going to be, and I see some default values here: https://github.com/guidance-ai/llguidance/blob/c83edb9527816dfaa8fbc61fe51384a5cba55e48/parser/src/api.rs#L251

Is it possible to tweak these values so I can allow longer running lexer jobs? In my case, I can wait if it means I get a good substring of a paper to use as evidence for something the LLM just generated, but right now it is just crashing almost straight away.

If this is exposed in python that would be great, but if not, do I have to resort to a manually installing llguidance from code with modified values?

Thanks for your help!

[hudson-ai]

I'm definitely of the opinion that we should expose these parameters, but I don't know whether the right way to do so is through guidance itself, environment variables, flags used when compiling llguidance from source (probably not that one), all of the above...?

@mmoskal @Harsha-Nori any opinions from you two? We've also discussed having higher defaults to make things work out-of-the-box for most users, where performance- and/or safety-critical deployments should reduce according to their needs.

Also @afg1 -- are you using guidance installed from pypi, or are you running on main these days? The substring implementation has been pushed down into rust, and guidance should now (fingers crossed) have some more sensible behaviors around how it decides what is or is not a lexeme. This may help you with some of these errors. We're hoping to have another release relatively soon, but if you can, do give main a try and report back!

[mmoskal]

I think it should go through Guidance itself (not sure where in the API). What limits are needed depends on the grammar, so env or compile options don't seem like a good choice.

Having said that, @afg1 please try out main and if you run into fuel issues there let us know. I believe it should work up to a megabyte or so.

[afg1]

Thanks for the info - I didn't think this had been exposed yet, look forward to being able to play with it!

On switching over to main - is it expected to 'just work'? I installed from main with pip install git+hhtps://... which I think does the trick. However, I'm now having a new issue which I think is llguidance specific - some tokens in Qwen's tokenizer are larger than 256 bytes so I'm hitting an assertion and crashing out. I may open an issue over there if that's something worthwhile?

In the meantime, I'll try using main with some other models to see if fuel is a problem still.

[hudson-ai]

@afg1 there's currently a bug in the way that some tokens from llamacpp are being converted to bytes... looking into it now.

But yeah, in the meantime, you should be able to do exactly that pip install (maybe upgrading llguidance if necessary) and give it a try on other models

[afg1]

Just to check back in - I used main after your fix for the token issue and I've had no problems with running out of fuel! I'm pretty confident it won't be an issue for me because the code I ran went for four days and processed about 42 million tokens without any (guidance related) problems.

Thanks for your help!

Edit: This work will be being written up shortly - what citation do you reccomend for guidance?

[hudson-ai]

@afg1 sorry for the delay -- something like this will be just fine, but we should really get something more "official" for you to cite at some point..!

@misc{guidance,
  author = {guidance-ai},
  title = {Guidance: A guidance language for controlling large language models},
  howpublished = {\url{https://github.com/guidance-ai/guidance}},
  note = {Accessed: 2025-04-10}
}