Skip to content

Retry-After headers when 'CreateChatCompletionStream' fails due rate limit hits #867

Open
@kanutron

Description

@kanutron

I believe this could be a duplicate of #452, but the discussion there diverged from the actual point.

Problem

When using the CreateChatCompletionStream, and hitting rate limits errors (Code: 429), the returned error nor the stream doesn't contain the http headers. I need to retrieve 'Retry-After' headers to pause the job for the specified time.

Those headers are standard and provide the amount of time (seconds/time-stamp on 'Retry-After', milliseconds on 'Retry-After-Ms').

If a 429 is obtained, what I do with CreateChatCompletion is to parse those headers and pause to go routine for the specified amount of time.

	resp, err := client.client.CreateChatCompletion(ctx, *openaiReq)
	if err != nil {
		... check status code ...
		parseAndPause(resp.Header()) // even with error, we still get `resp`
	}

But for CreateChatCompletionStream the above code could not work as:

	stream, err := client.client.CreateChatCompletionStream(ctx, *openaiReq)
	if err != nil {
		... check status code ...
		parseAndPause(stream.Header()) // stream.header could be nil, thus this panics
	}

Proposed solution

I believe here:

https://github.com/sashabaranov/go-openai/blob/7c145ebb4be68610bc3bb5377b754944307d44fd/client.go#L166C1-L172C3

	resp, err := client.config.HTTPClient.Do(req) //nolint:bodyclose // body is closed in stream.Close()
	if err != nil {
		return new(streamReader[T]), err
	}
	if isFailureStatusCode(resp) {
		return new(streamReader[T]), client.handleErrorResp(resp)
	}

Those returns should add the response into the streamReader struct.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions