bug: OpenAI API Endpoints not working (timeout) while Anthropic works fine

[1greentangerine]

Version: v0.8.2
Debian and Flathub

Describe the Bug

After activating the API server, only the Anthropic endpoint POST /v1/messages is working, the OpenAI style one POST /v1/chat/completions runs in a timeout

Steps to Reproduce

1. Download v0.8.2
2. Download a model
3. start API server
4. make the same request both the the anthropic and openai endpoint
5. openai endpoint will time out

Anthropic (works fine)

curl http://127.0.0.1:1337/v1/messages \
                          -H "Content-Type: application/json" \
                          -H "x-api-key: XXX" \
                          -d '{
                        "model": "Jan-v3.5-4B-Q4_K_XL",
                        "max_tokens": 1024,
                        "stream": true,
                        "messages": [{"role": "user", "content": "Hello!"}]
                      }'

OpenAI (will fail)

curl http://127.0.0.1:1337/v1/chat/completions \
                          -H "Content-Type: application/json" \
                          -H "Authorization: Bearer XXX" \
                          -d '{
                        "model": "Jan-v3.5-4B-Q4_K_XL",
                        "messages": [{"role": "user", "content": "Hello!"}],
                        "stream": true
                      }'

Screenshots / Logs

app.log

Operating System

• MacOS
• Windows
• Linux

[RailKill]

Likely related: Mintplex-Labs/anything-llm#5752

Guys at AnythingLLM reported that Jan only accepts minified JSON, but not pretty-printed JSON, which breaks OpenAI compatibility. So if you have whitespaces and newlines in your JSON, it will not work. This bug seems to have been introduced via the llama.cpp router in v0.8.0.