Skip to content

[Bug] legacy /generate's non-streaming disconnect branch discards its 400 response, returns 200 null instead #4781

Description

@AmirF194

Describe the bug

The non-streaming /generate endpoint (generate in lmdeploy/serve/openai/api_server.py) aborts the session correctly on client disconnect, but the caller never sees the 400 it computed: it gets HTTP 200 with a null body instead.

generate's inner helper returns the error response from inside an async with block:

response = None

async def _inner_call():
    ...
    async with aclosing(_with_request_cleanup(result_generator, [result_generator], [session])) as generator:
        async for res in generator:
            if await raw_request.is_disconnected():
                # Abort the request if the client disconnects.
                await session.async_abort()
                return create_error_response(HTTPStatus.BAD_REQUEST, 'Client disconnected')
            ...

    nonlocal response
    ...
    response = GenerateReqOutput(text=text, output_ids=output_ids, meta_info=meta)

await _inner_call()
return response

The call site is a bare await _inner_call(): its return value (the 400 create_error_response(...)) is discarded. response is a nonlocal initialized to None before the call and is only assigned inside the non-disconnect branch, after the async with exits normally. On disconnect, _inner_call returns early from inside the async with, so that assignment is never reached, and generate returns the still-None response instead of the 400.

session.async_abort() does fire correctly; only the response propagation is lost. This is the same discard-the-inner-return shape as #4776, in the sibling /generate endpoint instead of /v1/completions.

Reproduction

Docker (python:3.11-slim, CPU-only torch/transformers/etc., no GPU needed), current main (9140d372). Imported the real, unmodified generate, VariableInterface, GenerateReqInput, SessionManager, and GenOut, and drove the endpoint directly with a fake AsyncEngine.generate() that yields one item and a fake raw_request.is_disconnected() that returns True:

request = GenerateReqInput(prompt='hello', stream=False)
result = await generate(request, fake_raw_request)
assert isinstance(result, JSONResponse)  # fails: result is None

Environment

Reproduced against InternLM/lmdeploy at commit 9140d372bbf42209db66394a4268eeadaf05330c (2026-07-24), not tied to a specific GPU/OS environment (the fault is in request-handling logic, not a backend).

Error traceback

AssertionError: assert False
 +  where False = isinstance(None, <class 'starlette.responses.JSONResponse'>)

I have a fix ready (capture _inner_call's return value and return it directly when not None, mirroring how #4777 fixed the analogous /v1/completions case) and will open a PR referencing this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions