middleware concept

[Nayjest]

No description provided.

[github-actions]

I've Reviewed the Code

This PR adds a configurable middleware system to the LM proxy, enabling request/response interception for both streaming and non-streaming chat completions, but has a missing type import for TMiddleware in bootstrap.py and a potential race condition in the streaming middleware task handling that could cause unhandled exceptions or silent failures.

⚠️ 2 issues found across 4 files

#1 Missing TYPE_CHECKING import for TMiddleware

lm_proxy/bootstrap.py L17-L18

The type annotation list["TMiddleware"] references TMiddleware, but this type is not imported in the TYPE_CHECKING block. The existing code imports TLogger from .loggers but TMiddleware is missing, which will cause a NameError if type checking tools like mypy are used, or potentially at runtime in certain Python configurations.
Tags: bug, compatibility
Affected code:

17: if TYPE_CHECKING:
18:     from .loggers import TLogger

Proposed change:

if TYPE_CHECKING:
    from .loggers import TLogger
    from .middleware import TMiddleware

#2 Potential race condition: middleware_task.result() called without exception handling

lm_proxy/core.py L265-L267

At lines 266-267, after await asyncio.sleep(0), if middleware_task.done() is True and the task raised an exception, calling middleware_task.result() will re-raise that exception. However, this happens before returning the StreamingResponse, which could cause the entire request to fail even though the middleware failure may have been intentional early termination. More critically, if the task is NOT done (the common case), the middleware continues running but its exceptions won't be properly propagated since the task is fire-and-forget style.
Tags: bug, anti-pattern
Affected code:

265:             await asyncio.sleep(0)
266:             if middleware_task.done():
267:                 middleware_task.result()