`OpenAiService.ChatCompletion.CreateCompletion` Exception: The input does not contain any JSON tokens (Root cause: `HttpClient.Timeout`)

[chaojian-zhang]

Starting Words

Again, thank you very much for this handy library and sorry to have to bother you again🥲

Describe the bug
Occasionally when using the API, the service call itself will error out and cause an exception.
Unfortunately I don't know the shortest possible input to replicate this issue so please use the long version below.

Your code piece

var system = "In this setup, you are going to help the user to generate text-based contents according to their specifications.";
var user = """
In this setup, you are going to help me populate a database that represents personel information of a company. The database is going to be described using Prolog, specifically, the SWI implementation of Prolog.

See below snippet for schema and some information:

```prolog
human('Charles Wang', 55, male, 'Professional Engineer').
human('Alice Yu', 27, female, 'Software Developer').
human('Bob Jones', 30, male, 'Doctor').
human('Cynthia Lee', 25, female, 'Nurse').
human('David Smith', 5, male, 'Student').
father('Charles Wang', 'Alice Yu').
father('Charles Wang', 'Bob Jones').
mother('Cynthia Lee', 'David Smith').

father(X, Y) :- human(X, _, male, _), father(X, Y).
mother(X, Y) :- human(X, _, female, _), father(X, Y).
parent(X, Y) :- father(X, Y).
parent(X, Y) :- mother(X, Y).
son(X, Y) :- human(X, _, male, _), parent(Y, X).
daughter(X, Y) :- human(X, _, female, _), parent(Y, X).
grandparent(X, Y) :- parent(X, Z), parent(Z, Y).
grandfather(X, Y) :- grandparent(X, Y), human(X, _, male, _).
grandmother(X, Y) :- grandparent(X, Y), human(X, _, female, _).

company('Top Tech', 'Technology Consultation', 'Toronto', 150).
employee('Charles Wang', 'Top Tech', 2007, 125000).
employee('Thomas Ho', 'Top Tech', 2002, 140000).
employee('Sarah Adams', 'Top Tech', 2010, 100000).
employee('Jenny Smith', 'Top Tech', 2015, 90000).
teacher('William Brown', 'Top Tech', 2006, 110000).
manager('Thomas Ho', 'Charles Wang').
manager('Thomas Ho', 'Sarah Adams').
supervisor('Charles Wang', 'Jenny Smith').
supervisor('Sarah Adams', 'William Brown').
directly_reports_to(Subordinate, Manager) :- manager(Manager, Subordinate).
directly_reports_to(Subordinate, Supervisor) :- supervisor(Supervisor, Subordinate).
indirectly_reports_to(Subordinate, Manager) :- directly_reports_to(Subordinate, Supervisor), directly_reports_to(Supervisor, Manager).
team_member_of(EmployeeA, EmployeeB) :-
    directly_reports_to(EmployeeA, Manager),
    directly_reports_to(EmployeeB, Manager),
    EmployeeA \== EmployeeB.
```

Where:

1. `human` describes character name, age, gender and current profession/title.
2. `company` describes company name, company type, location, and number of employees.
3. `employee` describes employee name, employer name, start year, and current salary.

You may fix any syntax/grammar errors in existing code if necessary.
Now, help me populate a database with 25 employee information for a fictional VR hardware (technology) company named Dreamscape, where a majority of employee have an Asian background but most have adopted western first names; Other employees are local Canadians. The database should also populate all relative or related person information, i.e. people with direct relations to those 25 people, like parents, and children.
""";

Result

Run with:

OpenAIService OpenAiService = new OpenAIService(new OpenAiOptions()
{
    ApiKey = "Key......................................"
});
var CurrentModel = Models.Gpt_4;
var CurrentMaxTokenLimit = 8192;
var messages = new ChatMessage[]
{
    ChatMessage.FromSystem(system),
    ChatMessage.FromUser(user),
}
var completionResult = await OpenAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest()
{
    Messages = messages,
    Model = CurrentModel,
    MaxTokens = CurrentMaxTokenLimit - messages.Sum(m => m.Content.Length) - 21, /*MAX is 4096 including inputs and outputs; The additional 21 (sometimes 20) seems like the default input token size.*/
    Temperature = 1
});

And you will encounter a System.Text.Json.JsonException: 'The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. Path: $ | LineNumber: 0 | BytePositionInLine: 0.'

Expected behavior

Even if the service cannot handle it, it should not throw an exception - I thought that's what the completionResult.Successful should be checking.

Screenshots

Desktop (please complete the following information):

• OS: Windows 11
• Language: C# Net 7
• Version: Betalgo.OpenAI.GPT3 6.8.3

Additional context

Will upload shorter inputs that may trigger the same issue when available.

Complete Code

Copy and paste below code for a fresh Console Application created with Visual Studio targeting Net 7:

using OpenAI.GPT3.ObjectModels.RequestModels;
using OpenAI.GPT3.ObjectModels;
using OpenAI.GPT3.Managers;
using OpenAI.GPT3;

namespace ConsoleApp
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            var system = "In this setup, you are going to help the user to generate text-based contents according to their specifications.";
            var user = """
In this setup, you are going to help me populate a database that represents personel information of a company. The database is going to be described using Prolog, specifically, the SWI implementation of Prolog.

See below snippet for schema and some information:

```prolog
human('Charles Wang', 55, male, 'Professional Engineer').
human('Alice Yu', 27, female, 'Software Developer').
human('Bob Jones', 30, male, 'Doctor').
human('Cynthia Lee', 25, female, 'Nurse').
human('David Smith', 5, male, 'Student').
father('Charles Wang', 'Alice Yu').
father('Charles Wang', 'Bob Jones').
mother('Cynthia Lee', 'David Smith').

father(X, Y) :- human(X, _, male, _), father(X, Y).
mother(X, Y) :- human(X, _, female, _), father(X, Y).
parent(X, Y) :- father(X, Y).
parent(X, Y) :- mother(X, Y).
son(X, Y) :- human(X, _, male, _), parent(Y, X).
daughter(X, Y) :- human(X, _, female, _), parent(Y, X).
grandparent(X, Y) :- parent(X, Z), parent(Z, Y).
grandfather(X, Y) :- grandparent(X, Y), human(X, _, male, _).
grandmother(X, Y) :- grandparent(X, Y), human(X, _, female, _).

company('Top Tech', 'Technology Consultation', 'Toronto', 150).
employee('Charles Wang', 'Top Tech', 2007, 125000).
employee('Thomas Ho', 'Top Tech', 2002, 140000).
employee('Sarah Adams', 'Top Tech', 2010, 100000).
employee('Jenny Smith', 'Top Tech', 2015, 90000).
teacher('William Brown', 'Top Tech', 2006, 110000).
manager('Thomas Ho', 'Charles Wang').
manager('Thomas Ho', 'Sarah Adams').
supervisor('Charles Wang', 'Jenny Smith').
supervisor('Sarah Adams', 'William Brown').
directly_reports_to(Subordinate, Manager) :- manager(Manager, Subordinate).
directly_reports_to(Subordinate, Supervisor) :- supervisor(Supervisor, Subordinate).
indirectly_reports_to(Subordinate, Manager) :- directly_reports_to(Subordinate, Supervisor), directly_reports_to(Supervisor, Manager).
team_member_of(EmployeeA, EmployeeB) :-
    directly_reports_to(EmployeeA, Manager),
    directly_reports_to(EmployeeB, Manager),
    EmployeeA \== EmployeeB.
```

Where:

1. `human` describes character name, age, gender and current profession/title.
2. `company` describes company name, company type, location, and number of employees.
3. `employee` describes employee name, employer name, start year, and current salary.

You may fix any syntax/grammar errors in existing code if necessary.
Now, help me populate a database with 25 employee information for a fictional VR hardware (technology) company named Dreamscape, where a majority of employee have an Asian background but most have adopted western first names; Other employees are local Canadians. The database should also populate all relative or related person information, i.e. people with direct relations to those 25 people, like parents, and children.
""";
            var CurrentModel = Models.Gpt_4;
            var CurrentMaxTokenLimit = 8192;
            var messages = new ChatMessage[]
            {
                ChatMessage.FromSystem(system),
                ChatMessage.FromUser(user),
            };
            OpenAIService OpenAiService = new OpenAIService(new OpenAiOptions()
            {
                ApiKey = "Key......................................"
            });
            var completionResult = await OpenAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest()
            {
                Messages = messages,
                Model = CurrentModel,
                MaxTokens = CurrentMaxTokenLimit - messages.Sum(m => m.Content.Length) - 21, /*MAX is 4096 including inputs and outputs; The additional 21 (sometimes 20) seems like the default input token size.*/
                Temperature = 1
            });
        }
    }
}

[kayhantolga]

🙏🏻
It seems like you are getting an error which doesn't in Json format. Could you please try to capture incoming response?

You can do this easily with Laser Cat Eyes. It is a library (written by me and it is free to use)that shows you incoming and outgoing requests and responses. If you need help, let me know. Also, A sample usage is available in Playground Project.

I think the error is not related to the library. We can see more detail after you catch the response.

[chaojian-zhang]

By the way, with current execution speed of GPT-4, above code takes around 1:39 minutes to execute.

[chaojian-zhang]

🙏🏻 It seems like you are getting an error which doesn't in Json format. Could you please try to capture incoming response?

You can do this easily with Laser Cat Eyes. It is a library (written by me and it is free to use)that shows you incoming and outgoing requests and responses. If you need help, let me know. Also, A sample usage is available in Playground Project.

I think the error is not related to the library. We can see more detail after you catch the response.

1. How can it NOT be JSON when JSON is all OpenAI replies? (Ok, check my reply below to see comment - turns out it's indeed not a JSON)
2. How can this NOT be a library issue when the exception is raised INSIDE the library assembly?
3. As I mentioned, unless it's by design (which it cannot be since it's a .Net exception, not a Betalgo.OpenAI.GPT3 exception - there is no reason why the call to OpenAiService.ChatCompletion.CreateCompletion should raise an exception at all.

Anyway, I tried to make the same request in raw JSON REST request and here is the request and the response.

Request

{
	"model": "gpt-4",
	"messages": [
		{
			"role": "system",
			"content": "In this setup, you are going to help the user to generate text-based contents according to their specifications."
		},
		{
			"role": "user",
			"content": "In this setup, you are going to help me populate a database that represents personel information of a company. The database is going to be described using Prolog, specifically, the SWI implementation of Prolog.\r\n\r\nSee below snippet for schema and some information:\r\n\r\n```prolog\r\nhuman('Charles Wang', 55, male, 'Professional Engineer').\r\nhuman('Alice Yu', 27, female, 'Software Developer').\r\nhuman('Bob Jones', 30, male, 'Doctor').\r\nhuman('Cynthia Lee', 25, female, 'Nurse').\r\nhuman('David Smith', 5, male, 'Student').\r\nfather('Charles Wang', 'Alice Yu').\r\nfather('Charles Wang', 'Bob Jones').\r\nmother('Cynthia Lee', 'David Smith').\r\n\r\nfather(X, Y) :- human(X, _, male, _), father(X, Y).\r\nmother(X, Y) :- human(X, _, female, _), father(X, Y).\r\nparent(X, Y) :- father(X, Y).\r\nparent(X, Y) :- mother(X, Y).\r\nson(X, Y) :- human(X, _, male, _), parent(Y, X).\r\ndaughter(X, Y) :- human(X, _, female, _), parent(Y, X).\r\ngrandparent(X, Y) :- parent(X, Z), parent(Z, Y).\r\ngrandfather(X, Y) :- grandparent(X, Y), human(X, _, male, _).\r\ngrandmother(X, Y) :- grandparent(X, Y), human(X, _, female, _).\r\n\r\ncompany('Top Tech', 'Technology Consultation', 'Toronto', 150).\r\nemployee('Charles Wang', 'Top Tech', 2007, 125000).\r\nemployee('Thomas Ho', 'Top Tech', 2002, 140000).\r\nemployee('Sarah Adams', 'Top Tech', 2010, 100000).\r\nemployee('Jenny Smith', 'Top Tech', 2015, 90000).\r\nteacher('William Brown', 'Top Tech', 2006, 110000).\r\nmanager('Thomas Ho', 'Charles Wang').\r\nmanager('Thomas Ho', 'Sarah Adams').\r\nsupervisor('Charles Wang', 'Jenny Smith').\r\nsupervisor('Sarah Adams', 'William Brown').\r\ndirectly_reports_to(Subordinate, Manager) :- manager(Manager, Subordinate).\r\ndirectly_reports_to(Subordinate, Supervisor) :- supervisor(Supervisor, Subordinate).\r\nindirectly_reports_to(Subordinate, Manager) :- directly_reports_to(Subordinate, Supervisor), directly_reports_to(Supervisor, Manager).\r\nteam_member_of(EmployeeA, EmployeeB) :-\r\n    directly_reports_to(EmployeeA, Manager),\r\n    directly_reports_to(EmployeeB, Manager),\r\n    EmployeeA \\== EmployeeB.\r\n```\r\n\r\nWhere:\r\n\r\n1. `human` describes character name, age, gender and current profession\/title.\r\n2. `company` describes company name, company type, location, and number of employees.\r\n3. `employee` describes employee name, employer name, start year, and current salary.\r\n\r\nYou may fix any syntax\/grammar errors in existing code if necessary.\r\nNow, help me populate a database with 25 employee information for a fictional VR hardware (technology) company named Dreamscape, where a majority of employee have an Asian background but most have adopted western first names; Other employees are local Canadians. The database should also populate all relative or related person information, i.e. people with direct relations to those 25 people, like parents, and children."
		}
	]
}

Response

Conclusion

Ok, it turns out this is indeed NOT a JSON reply, my bad🥲

According to OpenAI: A Timeout error indicates that your request took too long to complete and our server closed the connection. This could be due to a network issue, a heavy load on our services, or a complex request that requires more processing time.

Again, I think this should be handled by the library, not by the user code.

Also, on above link it showed that:

However I don't know how I can change that parameter (and honestly, I shouldn't unless it's exposed by the library).

(EDIT: In Insomnia, one can disable timeout globally)

[chaojian-zhang]

I see this old issue, not sure about the strategy by the library author here: #186

[chaojian-zhang]

Some observations:

1. In other REST clients, e.g. Insomnia, one can explicitly set Timeout as global preferences (https://docs.insomnia.rest/insomnia/request-timeouts)
2. In .Net, one can set Timeout on System.Net.Http.HttpClient (https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.timeout?view=net-8.0)
   • Default value is only 100s
   • Since this is handled by the library and HttpClient is an instantiable object, I don't see a way for user to specify this
   • On the code (library) side, one might do this: https://stackoverflow.com/questions/10547895/how-can-i-tell-when-httpclient-has-timed-out

Also, just to prove that even set infinite time out this issue still happens:

[chaojian-zhang]

Hi, I can confirm that with one successful request that takes 109 seconds to complete using raw request in Insomnia, it cannot be done with current API because HttpClient.Timeout is by default set to 100 seconds.

The call to OpenAiService.ChatCompletion.CreateCompletion will raise an exception because it will automatically close the connection after 100 seconds (NOT OpenAI). This is apparently a library issue.

I saw this commit but the available options do not seem to allow configuring time out.

[chaojian-zhang]

For anyone else having the same problem, currently I am dealing with it with the following customized service provider (largely inspired by existing design of Betalgo OpenAI service):

using System;
using System.Collections.Generic;
using System.Net.Http.Headers;
using System.Net.Http;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Linq;
using Newtonsoft.Json;

namespace ChatService.Types
{
    public class ChatMessage
    {
        [JsonProperty("role")]
        public string Role { get; set; }
        [JsonProperty("content")]
        public string Content { get; set; }
        public ChatMessage(string role, string content)
        {
            Role = role;
            Content = content;
        }

        public static ChatMessage FromAssistant(string content)
        {
            return new ChatMessage("assistant", content);
        }
        public static ChatMessage FromUser(string content)
        {
            return new ChatMessage("user", content);
        }
        public static ChatMessage FromSystem(string content)
        {
            return new ChatMessage("system", content);
        }
    }

    public class ChatService
    {
        #region Construction
        public string APIKey { get; }
        public string Model { get; }
        public ChatService(string key, string model)
        {
            APIKey = key;
            Model = model;
        }
        #endregion

        #region Interface Method
        public async Task<string> Request(IEnumerable<ChatMessage> messages, int tokens, float temperature)
        {
            var inputJson = new InputJson
            {
                Model = Model,
                Messages = messages.ToArray()
            };

            var inputJsonString = JsonConvert.SerializeObject(inputJson);
            var content = new StringContent(inputJsonString, Encoding.UTF8, "application/json");

            using var client = new HttpClient();
            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", APIKey);
            client.Timeout = TimeSpan.FromMinutes(5); // Deal with longer execution time that's usually needed for GPT4

            var response = await client.PostAsync(EndpointUrl, content);

            if (response.StatusCode != HttpStatusCode.OK)
            {
                throw new Exception($"HTTP request failed with status code {response.StatusCode} and message {await response.Content.ReadAsStringAsync()}");
            }

            var responseJsonString = await response.Content.ReadAsStringAsync();
            var outputJson = JsonConvert.DeserializeObject<OutputJson>(responseJsonString);

            return outputJson.Choices.FirstOrDefault()?.Message.Content ?? null;
        }
        #endregion

        #region Implementation Details
        private readonly string EndpointUrl = "https://api.openai.com/v1/chat/completions";
        #endregion

        #region Types
        private class InputJson
        {
            [JsonProperty("model")]
            public string Model { get; set; }
            [JsonProperty("messages")]
            public ChatMessage[] Messages { get; set; }
        }
        private class OutputJson
        {
            [JsonProperty("id")]
            public string ID { get; set; }
            [JsonProperty("object")]
            public string Object { get; set; }
            [JsonProperty("model")]
            public string Model { get; set; }
            [JsonProperty("usage")]
            public Usage Usage { get; set; }
            [JsonProperty("choices")]
            public Choice[] Choices { get; set; }
        }
        private class Usage
        {
            [JsonProperty("prompt_tokens")]
            public string PromptTokens { get; set; }
            [JsonProperty("completion_tokens")]
            public string CompletionTokens { get; set; }
            [JsonProperty("total_tokens")]
            public string TotalTokens { get; set; }
        }
        private class Choice
        {
            [JsonProperty("message")]
            public ChatMessage Message { get; set; }
            [JsonProperty("finish_reason")]
            public string FinishReason { get; set; }
        }
        #endregion
    }
}

This works well for both GPT3.5-turbo and GPT4. For JSON serialization, it uses Newtonsoft.Json - you may replace it with Net 7 native JSON serializer.

[kayhantolga]

Hi, sorry. I was too busy last week and couldn't reply to any messages.

I know the library should handle HTML responses too but I don't have time for that yet. I appreciate it if you would like to send a PR fix that. Here are 2 samples if you would like to customize httpclient.

Since we both agree the main issue is not related to the library, I am going to move this thread to discussions.

If you don't agree or something else please let me know 🙏🏻

var openAI = new OpenAIService(new OpenAiOptions()
{
    //your options
},
    new HttpClient()
    {
        Timeout = TimeSpan.FromSeconds(240)
    });
    ```
    or
```serviceCollection.AddOpenAIService().ConfigureHttpClient( r=>r.Timeout = TimeSpan.FromSeconds(240));```