Optimized request prefill error messages

[learner0810]

fix: #637

The complete error message should be returned to the upstream rather than just an HTTP code.

• Before

~# curl -s http://inference-gateway-istio.lzj.svc.cluster.local:80/v1/chat/completions       -H "Content-Type: application/json"       -d '{
                "model": "Qwen/Qwen3-0.6B",
                "messages": [
                        {
                                "role": "user",
                                "content": [
                                        {
                                                "type": "text",
                                                "text": "Describe this image in one sentence."
                                        },
                                        {
                                                "type": "image_url",
                                                "image_url": {
                                                        "url": "https://cdn2.thecatapi.com/images/dui.jpg"
                                                }
                                        }
                                ]
                        }
                ]
        }'|jq .

• Now

curl -s http://inference-gateway-istio.lzj.svc.cluster.local:80/v1/chat/completions       -H "Content-Type: application/json"       -d '{
                "model": "Qwen/Qwen3-0.6B",
                "messages": [
                        {
                                "role": "user",
                                "content": [
                                        {
                                                "type": "text",
                                                "text": "Describe this image in one sentence."
                                        },
                                        {
                                                "type": "image_url",
                                                "image_url": {
                                                        "url": "https://cdn2.thecatapi.com/images/dui.jpg"
                                                }
                                        }
                                ]
                        }
                ]
        }'|jq .
{
  "error": {
    "code": 400,
    "message": "/model-cache/Qwen3-0.6B is not a multimodal model None",
    "param": null,
    "type": "BadRequestError"
  }
}

[Copilot]

Pull request overview

This PR enhances error handling in the NIXL Protocol V2 connector by returning complete error messages and headers to upstream clients when prefill requests fail, rather than just HTTP status codes.

Changes:

• Added header forwarding from prefill error responses to client responses
• Added response body writing to return detailed error messages
• Implemented proper error handling for write failures with logging

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[elevran]

The prefill request can fail for multiple reasons and returning an error here might not be optimal).

What do you think about approaching this in the following way instead:

• ignore prefill on failure (and not return its error to the caller) and
• process the request with the decode node (removing the KV cache transfer or any other P related artifacts from the request)?

Since P/D disaggregation is an optimization, this would sidestep any prefiller issues and (might?) still return a response to the user.
I think this would be a safer way to handle prefill errors/failures (e.g., when only the prefiller does not run the base model but the decoder does).

[learner0810]

I've adjusted some of the logic. Please take another look. If it's a client-side error, there's no need to forward the request to decode. In other cases, forward the request to decode.

[shmuelk]

A better overall solution might entail that we send multiple prefill targets in the "prefill header", if onbe fails we try another prefill node.

This is apparently very true with E/P/D, where the decode nodes are not allowed to do prefill.

@robertgshaw2-redhat

[robertgshaw2-redhat]

In the specific case I was looking at, the error will occur in every pod because the model name was incorrect in my input.

I think we should take a simplicity approach in error handling at the current stage of the feature

[elevran]

unrelated to this PR:
if there is an error, is the prefillerSpan reported and does it carry any valid and useful information back?
We might want to have the span close as error for observability.

/cc: @sallyom