Question about the temperature parameter in the Hugging Face Inference API

[xufengduan]

Hi everyone,

I have a question regarding the temperature parameter in the Hugging Face Inference API, particularly in the context of chat models. According to the documentation, the default value for temperature is 1. However, I noticed that some models seem to have a different default, such as 0.6, as specified in their generation_config.json file.

Here are my questions:

1. When using the Inference API, if I don’t explicitly set the temperature parameter, does the API always use the model’s default value from the generation_config.json? Or does it fall back to a global default of 1 as mentioned in the docs?

2. If I don’t pass in any additional parameters (like max_length, top_p, etc.), does the API automatically use all the defaults specified in the model’s generation_config.json file? Or are there other fallback defaults from the API side?

Thank you in advance for your help!

[fawern]

Q1: If you don't explicitly set the temperature, the Hugging Face Inference API will use the model's default value specified in its generation_config.json. It does not use a global default of 1 in this case.

Q2: If you don't provide these parameters, the API will use the defaults defined in the model's generation_config.json. If the model doesn't have those settings, the API may have its own fallback defaults.