[BUG] OpenAI SDK default timeout is not sufficient for chat completions, it should be configurable per operation

[lmolkova]

OpenAI operations can easily take a minute (based on my experiments), so the default response timeout of 60 sec is not enough.
OpenAI python SDK sets response timeout to 10 min (and connect timeout to 5 sec) - https://github.com/openai/openai-python/blob/main/src/openai/_constants.py#L9

[alzimmermsft]

Should we look into allowing the response and read timeouts to be mutated per-request to support this? 60 seconds is the default used by the HttpClient if nothing is configured.

[lmolkova]

great point. We should have a configurable timeout on RequestOptions- it makes sense to have a big timeout on specific APIs (e.g. chat completions) but not others (e.g. embeddings).

But also we should have reasonable defaults within OpenAI SDK (maybe also on per-API basis and at least for some APIs).

[Sanidhya3101]

Is there any update on the timeline of this?

[sken130]

Can we change the title of this issue to "Allow configurable timeout for each request"? Such as getChatCompletions()