fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459
Open
alvinttang wants to merge 1 commit into
Open
fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459alvinttang wants to merge 1 commit into
alvinttang wants to merge 1 commit into
Conversation
The factory function in guidance._grammar.subgrammar used a truthiness
check ("if temperature:") to decide whether to apply a temperature to
the generated RuleNode. This silently dropped the canonical greedy
setting temperature=0.0, meaning that
subgrammar(body, temperature=0.0)
produced a grammar with no temperature constraint at all — so sampling
was not forced to deterministic on the server side. The same truthiness
check was applied to max_tokens, which would have discarded max_tokens=0
as well (less impactful, but equally surprising).
Switch both checks to "is not None", matching the convention used by
guidance.json() and guidance._grammar.gen() throughout the codebase.
Regression tests walk the resulting grammar tree and confirm that
temperature=0.0 is reachable on a RuleNode.temperature field, and that
non-zero and None cases behave as expected.
ae79052 to
f49660d
Compare
riedgar-ms
approved these changes
May 14, 2026
riedgar-ms
left a comment
Collaborator
There was a problem hiding this comment.
I'm OK with this. @hudson-ai @nking-1 ?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
guidance._grammar.subgrammar()usedif temperature:/if max_tokens:to decide whether to attach those kwargs to the builtRuleNode. Because0.0and0are falsy in Python, callers passingtemperature=0.0(the canonical way to ask for deterministic / greedy sampling) ormax_tokens=0had their values silently dropped — the resulting grammar had noRuleNode.temperatureorRuleNode.max_tokensset.Every other factory in the codebase (
guidance.json,guidance._grammar.gen) already usesis not Nonefor the same kwargs, sosubgrammar()was the only inconsistent path.Root cause
Python treats
0and0.0as falsy, so the guards silently drop the valid-but-falsy values before they reach theRuleNode.Fix
Two-line change in
guidance/_grammar.py: swap both guards foris not None, matching the rest of the codebase:Tests
tests/unit/library/test_subgrammar.pyadds two parallel test classes:TestSubgrammarFactoryTemperature: verifiestemperature=0.0propagates (RED onmain),temperature=0.5still works,temperature=Nonesets nothing.TestSubgrammarFactoryMaxTokens: same three cases formax_tokens=0/max_tokens=16/max_tokens=None.A helper walks the resulting grammar tree via
GrammarNode.children()and collects everyRuleNode.temperature(ormax_tokens) that is notNone.Locally on
darwin/arm64:pytest tests/unit/library/test_subgrammar.py→ 14 passed.ruff checkclean.Risk notes
temperature=0.0ormax_tokens=0expecting them to be ignored will now see them forwarded tollguidance.llguidanceacceptsmax_tokens=0cleanly (verified locally viaLLMatchersmoke test) —max_tokens=0is also already the honoured value ingen()/json()/lark()per_ast.pyserialization, so this just bringssubgrammar()in line.None(the documented default) still means "unset" and sets no limit. That invariant is covered by the_none_sets_nothingtests.Related to the long-standing observation in #600 that temperature can be silently dropped on certain call paths; this PR addresses the
subgrammar()path specifically.