fix: handle Pydantic MockValSer bug in streaming responses (#18801)#24298
fix: handle Pydantic MockValSer bug in streaming responses (#18801)#24298AudreyKj wants to merge 1 commit intoBerriAI:mainfrom
Conversation
…8801) ## Problem TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer' when handling streaming responses with SAP AI Core and other providers. Pydantic 2.11+ has a bug where the internal MockValSer sentinel is not properly converted to a real SchemaSerializer in certain streaming scenarios. When LiteLLM tries to serialize chunks using model_dump(), it hits this corrupted serializer state. ## Solution Added try-catch fallback that uses __dict__ extraction when model_dump() fails with TypeError. This bypasses Pydantic's serialization entirely while maintaining functionality. ## Changes - litellm/litellm_core_utils/streaming_handler.py: Added fallback in 2 locations - litellm/litellm_core_utils/core_helpers.py: Added fallback in preserve_upstream_non_openai_attributes - tests/test_litellm/litellm_core_utils/test_streaming_handler.py: Added regression test ## Testing ✅ All 49 streaming handler tests pass ✅ Regression test verifies fallback behavior Related: BerriAI#18801 Related: pydantic/pydantic#7713
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds Key changes:
Issues found:
Confidence Score: 3/5
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/streaming_handler.py | Added try/except TypeError fallback around two model_dump() calls (lines 1862 and 2054) when stripping usage from streaming chunks; the broad catch may swallow unrelated TypeErrors and the __dict__ output is not structurally equivalent to model_dump() output. |
| litellm/litellm_core_utils/core_helpers.py | Added identical try/except TypeError fallback in preserve_upstream_non_openai_attributes; same broad-catch concern applies — any TypeError from model_dump() silently redirects to __dict__ extraction. |
| tests/test_litellm/litellm_core_utils/test_streaming_handler.py | New regression test validates the core_helpers fallback but does not exercise the two fallback paths inside __next__/__anext__ in streaming_handler.py; also contains an unused original_model_dump variable. |
Sequence Diagram
sequenceDiagram
participant Caller
participant CustomStreamWrapper
participant Pydantic
Caller->>CustomStreamWrapper: __next__() / __anext__()
CustomStreamWrapper->>Pydantic: response.model_dump()
alt Normal path (Pydantic ≤ 2.10 or no bug)
Pydantic-->>CustomStreamWrapper: obj_dict (fully serialized)
else MockValSer bug (Pydantic 2.11+)
Pydantic-->>CustomStreamWrapper: raises TypeError
CustomStreamWrapper->>CustomStreamWrapper: obj_dict = dict(response.__dict__)
end
CustomStreamWrapper->>CustomStreamWrapper: del obj_dict["usage"]
CustomStreamWrapper->>CustomStreamWrapper: model_response_creator(chunk=obj_dict)
CustomStreamWrapper-->>Caller: processed chunk (no usage)
Caller->>CustomStreamWrapper: return_processed_chunk_logic(...)
CustomStreamWrapper->>Pydantic: original_chunk.model_dump() [in preserve_upstream_non_openai_attributes]
alt Normal path
Pydantic-->>CustomStreamWrapper: obj_dict
else MockValSer bug
Pydantic-->>CustomStreamWrapper: raises TypeError
CustomStreamWrapper->>CustomStreamWrapper: obj_dict = dict(original_chunk.__dict__)
end
CustomStreamWrapper->>CustomStreamWrapper: setattr non-OpenAI fields onto model_response
CustomStreamWrapper-->>Caller: model_response
Last reviewed commit: "fix: handle Pydantic..."
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| except TypeError as e: | ||
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | ||
| obj_dict = dict(response.__dict__) if hasattr(response, '__dict__') else {} |
There was a problem hiding this comment.
TypeError catch swallows unrelated errors
The fallback catches every TypeError, not just the MockValSer one. If model_dump() raises a TypeError for a different reason (e.g., a genuine type mismatch in a custom serializer or a programming mistake), the code will silently fall back to __dict__, potentially returning subtly wrong/incomplete data instead of surfacing the real bug.
Additionally, the __dict__ of a Pydantic v2 model and the output of model_dump() are not equivalent: model_dump() recursively serializes nested models to plain dicts/primitives, while __dict__ returns the raw Python objects (nested Pydantic model instances, enums, etc.). Passing this mixed-type dict to model_response_creator could produce unexpected results depending on how the creator handles nested objects.
Consider narrowing the guard to only the known error string:
| except TypeError as e: | |
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | |
| obj_dict = dict(response.__dict__) if hasattr(response, '__dict__') else {} | |
| try: | |
| obj_dict = response.model_dump() | |
| except TypeError as e: | |
| if "MockValSer" not in str(e): | |
| raise | |
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | |
| obj_dict = dict(response.__dict__) if hasattr(response, '__dict__') else {} |
Same pattern applies to the equivalent catch at line 2056 and in core_helpers.py.
| except TypeError as e: | ||
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | ||
| obj_dict = dict(processed_chunk.__dict__) if hasattr(processed_chunk, '__dict__') else {} |
There was a problem hiding this comment.
e
The exception is captured as e but never used or logged. This silently discards the exception information, making future debugging harder if a different TypeError accidentally gets swallowed here.
The same issue exists at line 1864.
| except TypeError as e: | |
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | |
| obj_dict = dict(processed_chunk.__dict__) if hasattr(processed_chunk, '__dict__') else {} | |
| except TypeError: | |
| # Fallback: manually extract dict from __dict__ to bypass Pydantic serializer | |
| obj_dict = dict(processed_chunk.__dict__) if hasattr(processed_chunk, '__dict__') else {} |
| ) | ||
|
|
||
| # Mock model_dump to raise TypeError (simulating MockValSer bug) | ||
| original_model_dump = chunk_with_usage.model_dump |
There was a problem hiding this comment.
original_model_dump
original_model_dump is assigned but never referenced again. It was likely intended for restoring the method after the test (to avoid leaking the mock), but the cleanup was omitted.
If this is intentional, remove the assignment. If teardown was intended, add it:
try:
result = initialized_custom_stream_wrapper.return_processed_chunk_logic(...)
finally:
chunk_with_usage.model_dump = original_model_dump| # Process the chunk through return_processed_chunk_logic which calls model_dump | ||
| result = initialized_custom_stream_wrapper.return_processed_chunk_logic( | ||
| completion_obj={"content": "test content"}, | ||
| response_obj={"original_chunk": chunk_with_usage}, | ||
| model_response=chunk_with_usage, | ||
| ) | ||
|
|
||
| # Should not raise TypeError and should successfully process the chunk | ||
| assert result is not None | ||
| assert result.choices[0].delta.content == "test content" |
There was a problem hiding this comment.
streaming_handler.py fallback paths
The test exercises return_processed_chunk_logic, which invokes the fallback in core_helpers.preserve_upstream_non_openai_attributes. However, the two new except TypeError blocks added in streaming_handler.py (lines 1862–1866 and 2054–2058) live inside the synchronous __next__ and asynchronous __anext__ iterators respectively — neither of which is called by return_processed_chunk_logic.
As a result, the regression test does not actually verify the two most direct code paths changed by this PR. Consider adding a test that iterates the wrapper (e.g., via list(wrapper) or async for chunk in wrapper) with model_dump patched to raise, to confirm those paths also survive the bug.
1 participant
Problem
TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer'when handling streaming responses with SAP AI Core and other providers (vLLM, etc).Related GitHub Issue: #18801
Related Pydantic Issue: pydantic/pydantic#7713
Root Cause
Pydantic 2.11+ has a bug where the internal
MockValSersentinel is not properly converted to a realSchemaSerializerin certain streaming scenarios. When LiteLLM tries to serialize streaming chunks usingmodel_dump(), it hits this corrupted serializer state and crashes.The bug occurs when:
model_dump()calls crash with MockValSer TypeErrorSolution
Added try-catch fallback that uses
__dict__extraction whenmodel_dump()fails withTypeError. This bypasses Pydantic's broken serialization entirely while maintaining all functionality.The fix is:
Changes
Modified Files
litellm/litellm_core_utils/streaming_handler.py
model_dump()is called on streaming chunkslitellm/litellm_core_utils/core_helpers.py
preserve_upstream_non_openai_attributes()function (~line 273)tests/test_litellm/litellm_core_utils/test_streaming_handler.py
test_model_dump_fallback_handles_pydantic_serializer_bugTesting
✅ All 49 streaming handler tests pass
python3 -m pytest tests/test_litellm/litellm_core_utils/test_streaming_handler.py -v # 49 passed, 70 warnings in 3.65s✅ Regression test verifies fallback behavior
model_dump()to raise MockValSer TypeError__dict__extraction works correctlyWhy Not Alternative Solutions?
mode='python': Still uses the broken serializer internally__dict__fallback: Bypasses serialization entirely, works immediatelyImpact
This fix resolves streaming issues for:
Users experiencing the
MockValSererror will now have streaming work correctly without any configuration changes.Checklist
tests/test_litellm/litellm_core_utils/test_streaming_handler.pymake test-unit-core-utilsequivalent)