[BUG]max_new_tokens do not work

[Er-gou123]

Describe the bug

llm = OpenAILLM(
model="deepseek-reasoner",
base_url="https://api.deepseek.com/v1",
api_key="",
generation_kwargs={
"temperature": 0.7,
"max_new_tokens": 2048,
},
timeout=60,
max_retries=3,
)
the max_new_tokens is also 128(default)

output = llm.generate_outputs(inputs=inputs, max_new_tokens=1000)
it work in generate_outputs

To reproduce

No response

Expected behavior

No response

Screenshots

No response

Environment

• Distilabel Version [e.g. 1.0.0]:
• Python Version [e.g. 3.11]:

Additional context

No response

[dameikle]

Hi @Er-gou123 - should this not be max_tokens for the DeepSeek API?
https://api-docs.deepseek.com/api/create-chat-completion#request

[Er-gou123]

yes,it just does not work in code.

from openai import OpenAI
from distilabel.llms import OpenAILLM
import logging
logging.basicConfig(level=logging.DEBUG)

prompt_template = """\

You will be given a problem. Please reason step by step, and put your final answer within \boxed{}:

{{ instruction }}"""

llm = OpenAILLM(
model="deepseek-reasoner",
base_url="https://api.deepseek.com/v1",
api_key="",

timeout=60,
generation_kwargs={
    "temperature": 0.7,
    "max_new_tokens": 2048,
},
max_retries=3,

)

llm.load()

inputs = [[{"role": "user", "content": "Hello, who are you"}]]

output = llm.generate_outputs(inputs=inputs)
print("output:", output)

Here is my code, and below is my code log.

DEBUG:asyncio:Using selector: EpollSelector
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'idempotency_key': 'stainless-python-retry-35fb3d73-6d1f-4e75-8f69-0f738fab4187', 'json_data': {'messages': [{'role': 'user', 'content': 'Hello, who are you'}], 'model': 'deepseek-reasoner', 'frequency_penalty': 0.0, 'logprobs': False, 'max_tokens': 128, 'n': 1, 'presence_penalty': 0.0, 'stop': None, 'temperature': 1.0, 'top_logprobs': None, 'top_p': 1.0}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.deepseek.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.deepseek.com' port=443 local_address=None timeout=60 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x72e9f5b87bd0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x72e9f156c710> server_hostname='api.deepseek.com' timeout=60
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x72e9f3323c50>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Date', b'Wed, 02 Jul 2025 15:27:42 GMT'), (b'Content-Type', b'application/json'), (b'Transfer-Encoding', b'chunked'), (b'Connection', b'keep-alive'), (b'Set-Cookie', b'HWWAFSESID=2fed273b6dd9c6f835; path=/'), (b'Set-Cookie', b'HWWAFSESTIME=1751470062549; path=/'), (b'vary', b'origin, access-control-request-method, access-control-request-headers'), (b'access-control-allow-credentials', b'true'), (b'x-ds-trace-id', b'e90c3c13a6829c7b8dea32529c8b916d'), (b'Strict-Transport-Security', b'max-age=31536000; includeSubDomains; preload'), (b'X-Content-Type-Options', b'nosniff'), (b'Server', b'CW'), (b'Content-Encoding', b'gzip')])
INFO:httpx:HTTP Request: POST https://api.deepseek.com/v1/chat/completions "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:openai._base_client:HTTP Response: POST https://api.deepseek.com/v1/chat/completions "200 OK" Headers([('date', 'Wed, 02 Jul 2025 15:27:42 GMT'), ('content-type', 'application/json'), ('transfer-encoding', 'chunked'), ('connection', 'keep-alive'), ('set-cookie', 'HWWAFSESID=2fed273b6dd9c6f835; path=/'), ('set-cookie', 'HWWAFSESTIME=1751470062549; path=/'), ('vary', 'origin, access-control-request-method, access-control-request-headers'), ('access-control-allow-credentials', 'true'), ('x-ds-trace-id', 'e90c3c13a6829c7b8dea32529c8b916d'), ('strict-transport-security', 'max-age=31536000; includeSubDomains; preload'), ('x-content-type-options', 'nosniff'), ('server', 'CW'), ('content-encoding', 'gzip')])
DEBUG:openai._base_client:request_id: None

Take a close look at the log:
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'idempotency_key': 'stainless-python-retry-35fb3d73-6d1f-4e75-8f69-0f738fab4187', 'json_data': {'messages': [{'role': 'user', 'content': 'Hello, who are you'}], 'model': 'deepseek-reasoner', 'frequency_penalty': 0.0, 'logprobs': False, 'max_tokens': 128, 'n': 1, 'presence_penalty': 0.0, 'stop': None, 'temperature': 1.0, 'top_logprobs': None, 'top_p': 1.0}}
The parameter 'max_tokens': 128 has not been changed to 2048.

[dameikle]

I see what you mean, it looks like the generation_kwarg runtime param is not able to be passed when constructing. It would be nice to allow that similar to the Huggingface Inference Endpoints.

I don't have DeepSeek access but can pass it directly this way for OpenAI for your example:
`from distilabel.models import OpenAILLM

llm = OpenAILLM(model="gpt-4-turbo",api_key="")
llm.load()

inputs = [[{"role": "user", "content": "Write a blog post about sites to see in London"}]]
output = llm.generate_outputs(inputs=inputs, max_new_tokens=2048)

print(output)`

I quite often set it as a parameter on the pipeline:

`from datasets import load_dataset

from distilabel.models import OpenAILLM
from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration

with Pipeline() as pipeline:
text = TextGeneration(
llm=OpenAILLM(
model="gpt-4-turbo",
api_key="<key_here>",
),
)

if name == "main":
dataset = load_dataset("distilabel-internal-testing/instructions", split="test")
distiset = pipeline.run(
dataset=dataset,
parameters={
text.name: {
"llm": {"generation_kwargs": {"max_new_tokens": 4096}}
}
},
)
distiset.push_to_hub(repo_id="davidmeikle/distilabel-example-max-tokens-test")`