Skip to content

Preserve unlimited token sentinel in text generation CLI - #269

Open
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/generate-unlimited-limit
Open

Preserve unlimited token sentinel in text generation CLI#269
sylvesterkaczmarek wants to merge 2 commits into
openai:mainfrom
sylvesterkaczmarek:fix/generate-unlimited-limit

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

Pass the generation CLI's documented unlimited-token sentinel to the selected backend unchanged.

--limit 0 is the default and is documented as disabling the token limit, but gpt_oss.generate currently converts 0 to None. The Torch and Triton generators implement unlimited generation with integer 0 and compare num_generated_tokens < max_tokens, so None can cause a TypeError. The vLLM generator also accepts 0 and converts it internally for vLLM.

Fixes #268.

Fix

Pass args.limit directly as max_tokens.

Regression coverage

Adds a backend-stub test around main() verifying that the default/unlimited limit=0 reaches the token generator as integer 0 without requiring a checkpoint, GPU, or inference dependency.

Finite positive limits, stop tokens, temperature, logprob behavior, and backend selection are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Text generation default unlimited limit passes an invalid max_tokens value

1 participant