Unable to get message content from chat functionality

[achenbarrett]

Hi All,

I'm trying to use a custom endpoint with the extension where my custom endpoint will take the chat messages from the extension, forward them to an LLM service provider, and then send the response from the LLM provider back to the extension. To test for now, my custom endpoint is a simple Flask app running on localhost. The Flask app routes the /chat/completions post request, authenticates using the api key in the request header, sends the body of the request to an Azure OpenAI endpoint as the messages payload, and then returns the Azure OpenAI endpoint ChatCompletions object as a json response.

I receive the request payload from the extension and am able to successfully parse the body of the request to send to Azure OpenAI. The Azure OpenAI service successfully sends me back a ChatCompletions object and that is successfully returned to the extension as JSON, however the messages content does not appear in the chat window. I can see in the extension output window that the correct number of prompt tokens, completion tokens, total tokens, and session tokens are getting parsed out from the response I am returning to the extension. As far as I can tell, the extension is simply using the TypeScript SDK completions API which would expect back a JSON response to be able to build the ChatCompletions object and the ChatCompletions object is then used to get the mssage content and token counts which are rendered in the extension's chat window. Am I missing something here? Is there a specific format I need to be sending responses back to the extension to have them properly parsed and displayed?

Here's the flask app that I'm using as the custom endpoint for reference. I stripped out the authentication stuff in the flask app since it's not relevant to the problem I'm facing here.

from dotenv import load_dotenv
from openai import AzureOpenAI
from quart import Quart, jsonify, request

api_version = "2024-03-01-preview"
AZURE_OPENAI_ENDPOINT = "https://some/valid/azure/openai/endpoint"

load_dotenv()
azure_openai_client = AzureOpenAI(
    api_version=api_version,
    azure_endpoint=AZURE_OPENAI_ENDPOINT,
    api_key=os.environ.get("OPENAI_API_KEY"),
)

app = Quart(__name__)


@app.route("/chat/completions", methods=["POST"])
async def chat():
    # Build prompt.
    data = await request.get_json()

    try:
        # Instantiate the chat coroutine.
        response = azure_openai_client.chat.completions.create(
            model="gpt-4",
            messages=data["messages"],
            seed=56,
            temperature=0.7,  # temperature for assistant mode.
            max_tokens=1200,  # token limit for chat.
            n=1,
            stream=False,  # hardcode for simplicity.
        )
        # Return a completion response.
        result = response.to_dict()
        return jsonify(result)

    except Exception as e:
        return jsonify({"error": str(e)}), 500


if __name__ == "__main__":
    app.run(debug=True)

And here is an example of the OpenAI ChatCompletions response that I am sending back to the extension as JSON:

{
    "id": "chatcmpl-123",
    "object": "chat.completion",
    "created": 1677652288,
    "model": "gpt-4o-mini",
    "system_fingerprint": "fp_44709d6fcb",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "\n\nHello there, how may I assist you today?",
            },
            "content_filter_results": {
                "hate": {"filtered": False, "severity": "safe"},
                "self_harm": {"filtered": False, "severity": "safe"},
                "sexual": {"filtered": False, "severity": "safe"},
                "violence": {"filtered": False, "severity": "safe"},
            },
            # "logprobs": null,
            "finish_reason": "stop",
        }
    ],
    "usage": {
        "prompt_tokens": 9,
        "completion_tokens": 12,
        "total_tokens": 21,
        "completion_tokens_details": {"reasoning_tokens": 0},
    },
    "prompt_filter_results": [
        {
            "prompt_index": 0,
            "content_filter_results": {
                "hate": {"filtered": False, "severity": "safe"},
                "self_harm": {"filtered": False, "severity": "safe"},
                "sexual": {"filtered": False, "severity": "safe"},
                "violence": {"filtered": False, "severity": "safe"},
            },
        }
    ],
}