Skip to content

fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459

Open
alvinttang wants to merge 1 commit into
guidance-ai:mainfrom
alvinttang:fix/subgrammar-zero-temperature
Open

fix: preserve temperature=0.0 and max_tokens=0 in subgrammar() factory#1459
alvinttang wants to merge 1 commit into
guidance-ai:mainfrom
alvinttang:fix/subgrammar-zero-temperature

Conversation

@alvinttang

Copy link
Copy Markdown

Summary

guidance._grammar.subgrammar() used if temperature: / if max_tokens: to decide whether to attach those kwargs to the built RuleNode. Because 0.0 and 0 are falsy in Python, callers passing temperature=0.0 (the canonical way to ask for deterministic / greedy sampling) or max_tokens=0 had their values silently dropped — the resulting grammar had no RuleNode.temperature or RuleNode.max_tokens set.

Every other factory in the codebase (guidance.json, guidance._grammar.gen) already uses is not None for the same kwargs, so subgrammar() was the only inconsistent path.

Root cause

# guidance/_grammar.py (before)
if max_tokens:          # wrong: 0 is valid
    body = token_limit(body, max_tokens=max_tokens)
if temperature:         # wrong: 0.0 is valid
    body = with_temperature(body, temperature=temperature)

Python treats 0 and 0.0 as falsy, so the guards silently drop the valid-but-falsy values before they reach the RuleNode.

Fix

Two-line change in guidance/_grammar.py: swap both guards for is not None, matching the rest of the codebase:

if max_tokens is not None:
    body = token_limit(body, max_tokens=max_tokens)
if temperature is not None:
    body = with_temperature(body, temperature=temperature)

Tests

tests/unit/library/test_subgrammar.py adds two parallel test classes:

  • TestSubgrammarFactoryTemperature: verifies temperature=0.0 propagates (RED on main), temperature=0.5 still works, temperature=None sets nothing.
  • TestSubgrammarFactoryMaxTokens: same three cases for max_tokens=0 / max_tokens=16 / max_tokens=None.

A helper walks the resulting grammar tree via GrammarNode.children() and collects every RuleNode.temperature (or max_tokens) that is not None.

Locally on darwin/arm64:

  • pytest tests/unit/library/test_subgrammar.py → 14 passed.
  • Full unit suite → 1582 passed, 1 skipped, 220 xfailed (all pre-existing; no new failures).
  • ruff check clean.

Risk notes

  • Behaviour change: callers who were passing temperature=0.0 or max_tokens=0 expecting them to be ignored will now see them forwarded to llguidance. llguidance accepts max_tokens=0 cleanly (verified locally via LLMatcher smoke test) — max_tokens=0 is also already the honoured value in gen() / json() / lark() per _ast.py serialization, so this just brings subgrammar() in line.
  • None (the documented default) still means "unset" and sets no limit. That invariant is covered by the _none_sets_nothing tests.
  • No API shape change.

Related to the long-standing observation in #600 that temperature can be silently dropped on certain call paths; this PR addresses the subgrammar() path specifically.

The factory function in guidance._grammar.subgrammar used a truthiness
check ("if temperature:") to decide whether to apply a temperature to
the generated RuleNode. This silently dropped the canonical greedy
setting temperature=0.0, meaning that

    subgrammar(body, temperature=0.0)

produced a grammar with no temperature constraint at all — so sampling
was not forced to deterministic on the server side. The same truthiness
check was applied to max_tokens, which would have discarded max_tokens=0
as well (less impactful, but equally surprising).

Switch both checks to "is not None", matching the convention used by
guidance.json() and guidance._grammar.gen() throughout the codebase.

Regression tests walk the resulting grammar tree and confirm that
temperature=0.0 is reachable on a RuleNode.temperature field, and that
non-zero and None cases behave as expected.
@alvinttang alvinttang force-pushed the fix/subgrammar-zero-temperature branch from ae79052 to f49660d Compare April 24, 2026 01:06

@riedgar-ms riedgar-ms left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm OK with this. @hudson-ai @nking-1 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants