Describe the bug
The non-streaming /generate endpoint (generate in lmdeploy/serve/openai/api_server.py) aborts the session correctly on client disconnect, but the caller never sees the 400 it computed: it gets HTTP 200 with a null body instead.
generate's inner helper returns the error response from inside an async with block:
response = None
async def _inner_call():
...
async with aclosing(_with_request_cleanup(result_generator, [result_generator], [session])) as generator:
async for res in generator:
if await raw_request.is_disconnected():
# Abort the request if the client disconnects.
await session.async_abort()
return create_error_response(HTTPStatus.BAD_REQUEST, 'Client disconnected')
...
nonlocal response
...
response = GenerateReqOutput(text=text, output_ids=output_ids, meta_info=meta)
await _inner_call()
return response
The call site is a bare await _inner_call(): its return value (the 400 create_error_response(...)) is discarded. response is a nonlocal initialized to None before the call and is only assigned inside the non-disconnect branch, after the async with exits normally. On disconnect, _inner_call returns early from inside the async with, so that assignment is never reached, and generate returns the still-None response instead of the 400.
session.async_abort() does fire correctly; only the response propagation is lost. This is the same discard-the-inner-return shape as #4776, in the sibling /generate endpoint instead of /v1/completions.
Reproduction
Docker (python:3.11-slim, CPU-only torch/transformers/etc., no GPU needed), current main (9140d372). Imported the real, unmodified generate, VariableInterface, GenerateReqInput, SessionManager, and GenOut, and drove the endpoint directly with a fake AsyncEngine.generate() that yields one item and a fake raw_request.is_disconnected() that returns True:
request = GenerateReqInput(prompt='hello', stream=False)
result = await generate(request, fake_raw_request)
assert isinstance(result, JSONResponse) # fails: result is None
Environment
Reproduced against InternLM/lmdeploy at commit 9140d372bbf42209db66394a4268eeadaf05330c (2026-07-24), not tied to a specific GPU/OS environment (the fault is in request-handling logic, not a backend).
Error traceback
AssertionError: assert False
+ where False = isinstance(None, <class 'starlette.responses.JSONResponse'>)
I have a fix ready (capture _inner_call's return value and return it directly when not None, mirroring how #4777 fixed the analogous /v1/completions case) and will open a PR referencing this issue.
Describe the bug
The non-streaming
/generateendpoint (generateinlmdeploy/serve/openai/api_server.py) aborts the session correctly on client disconnect, but the caller never sees the400it computed: it getsHTTP 200with anullbody instead.generate's inner helper returns the error response from inside anasync withblock:The call site is a bare
await _inner_call(): its return value (the400create_error_response(...)) is discarded.responseis anonlocalinitialized toNonebefore the call and is only assigned inside the non-disconnect branch, after theasync withexits normally. On disconnect,_inner_callreturns early from inside theasync with, so that assignment is never reached, andgeneratereturns the still-Noneresponseinstead of the400.session.async_abort()does fire correctly; only the response propagation is lost. This is the same discard-the-inner-return shape as #4776, in the sibling/generateendpoint instead of/v1/completions.Reproduction
Docker (
python:3.11-slim, CPU-onlytorch/transformers/etc., no GPU needed), currentmain(9140d372). Imported the real, unmodifiedgenerate,VariableInterface,GenerateReqInput,SessionManager, andGenOut, and drove the endpoint directly with a fakeAsyncEngine.generate()that yields one item and a fakeraw_request.is_disconnected()that returnsTrue:Environment
Reproduced against
InternLM/lmdeployat commit9140d372bbf42209db66394a4268eeadaf05330c(2026-07-24), not tied to a specific GPU/OS environment (the fault is in request-handling logic, not a backend).Error traceback
I have a fix ready (capture
_inner_call's return value and return it directly when notNone, mirroring how #4777 fixed the analogous/v1/completionscase) and will open a PR referencing this issue.