BUG: Follower is not able to parse error responses from leader if an invalid request is sent. #1154
Description
What happened?
If a user or client sends a mutation request (like adding new schema, change compatibility level etc) to Karapace which has a leader-follower deployment to a follower, that request is forwarded to the leader which is expected. But if the leader throws an error such as 409 or other error codes as defined, the follower is not parsing it correctly and instead returns "Internal Server Error" which is a plaintext response. This is causing Confluent clients to log errors as it expects a FastAPI response.
Currently only valid requests which return 200 or some error responses are being handled correctly by the follower.
What did you expect to happen?
We expect that the error message to be properly parsed by the follower and sent back to the client.
What else do we need to know?
This is printing stack trace in the follower pod logs which could expose sensitive info and accumulation of these logs. Also this causes the follower to respond with "Internal Server Error" even though the leader has handled the error.
More details to reproduce this error are given below:
We have currently deployed karapace with a leader and follower pod as a StatefulSet in k8s:
kubectl get po -n schema-registry
NAME READY STATUS RESTARTS AGE
karapace-schema-registry-0 1/1 Running 0 8d
karapace-schema-registry-1 1/1 Running 0 8d
//query health endpoint to get leader
GET http://karapace-schema-registry.schema-registry:8081/_health
Response:
{
"karapace_version": "5.0.2",
"status": {
"schema_registry_ready": true,
"schema_registry_startup_time_sec": 6.625380557961762,
"schema_registry_reader_current_offset": 493,
"schema_registry_reader_highest_offset": 493,
"schema_registry_is_primary": true,
"schema_registry_is_primary_eligible": true,
"schema_registry_primary_url": "http://karapace-schema-registry-0.karapace-schema-registry.schema-registry:8081",
"schema_registry_coordinator_running": true,
"schema_registry_coordinator_generation_id": 6
},
"healthy": true
}
As per the above request, http://karapace-schema-registry-0.karapace-schema-registry.schema-registry:8081 is the leader's url.
Now , we will try to change the compatibility mode which is invalid like below to the leader:
PUT http://karapace-schema-registry-0.karapace-schema-registry.schema-registry:8081/config
Request body:
{"compatibility":"RANDOM STRING"}
Response: 422 Unprocessable Entity
{
"error_code": 42203,
"message": "Invalid compatibility level. Valid values are none, backward, forward, full, backward_transitive, forward_transitive, and full_transitive"
}
This is the expected response. Now try the same request one or more times to the follower with it's url as shown:
PUT http://karapace-schema-registry-1.karapace-schema-registry.schema-registry:8081/config
Request body:
{"compatibility":"RANDOM STRING"}
Response: 500 Internal Server Error
Internal Server Error
Follower pod logs:
kubectl logs karapace-schema-registry-1 -n schema-registry
uvicorn.error MainThread ERROR Exception in ASGI application
- Exception Group Traceback (most recent call last):
| File "/venv/lib/python3.10/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 186, in call
| async with anyio.create_task_group() as task_group:
| File "/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit
| raise BaseExceptionGroup(
| exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| File "/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in call
| await self.middleware_stack(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in call
| with collapse_excgroups():
| File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
| self.gen.throw(typ, value, traceback)
| File "/venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in call
| response = await self.dispatch_func(request, call_next)
| File "/venv/lib/python3.10/site-packages/dependency_injector/wiring.py", line 994, in _patched
| return await _async_inject(
| File "src/dependency_injector/_cwiring.pyx", line 66, in _async_inject
| File "/venv/lib/python3.10/site-packages/karapace/api/telemetry/middleware.py", line 39, in telemetry_middleware
| raise exc
| File "/venv/lib/python3.10/site-packages/karapace/api/telemetry/middleware.py", line 32, in telemetry_middleware
| response: Response = await call_next(request)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
| raise app_exc
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in call
| with collapse_excgroups():
| File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
| self.gen.throw(typ, value, traceback)
| File "/venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in call
| response = await self.dispatch_func(request, call_next)
| File "/venv/lib/python3.10/site-packages/karapace/api/middlewares/init.py", line 77, in set_content_types
| response = await call_next(request)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
| raise app_exc
| File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
| await self.app(scope, receive_or_disconnect, send_no_error)
| File "/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in call
| await self.middleware_stack(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
| await route.handle(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
| response = await f(request)
| File "/venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
| raw_response = await run_endpoint_function(
| File "/venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
| return await dependant.call(**values)
| File "/venv/lib/python3.10/site-packages/dependency_injector/wiring.py", line 994, in _patched
| return await _async_inject(
| File "src/dependency_injector/_cwiring.pyx", line 66, in _async_inject
| File "/venv/lib/python3.10/site-packages/karapace/api/routers/config.py", line 62, in config_put
| return await forward_client.forward_request_remote(
| File "/venv/lib/python3.10/site-packages/karapace/api/forward_client.py", line 105, in forward_request_remote
| return response_type.parse_raw(body) # type: ignore[return-value]
| File "/venv/lib/python3.10/site-packages/pydantic/main.py", line 1277, in parse_raw
| return cls.model_validate(obj)
| File "/venv/lib/python3.10/site-packages/pydantic/main.py", line 627, in model_validate
| return cls.pydantic_validator.validate_python(
| pydantic_core._pydantic_core.ValidationError: 1 validation error for CompatibilityResponse
| compatibility
| Field required [type=missing, input_value={'error_code': 42203, 'me...e, and full_transitive'}, input_type=dict]
| For further information visit https://errors.pydantic.dev/2.10/v/missing
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "/venv/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in call
with collapse_excgroups():
File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in call
response = await self.dispatch_func(request, call_next)
File "/venv/lib/python3.10/site-packages/dependency_injector/wiring.py", line 994, in _patched
return await _async_inject(
File "src/dependency_injector/_cwiring.pyx", line 66, in _async_inject
File "/venv/lib/python3.10/site-packages/karapace/api/telemetry/middleware.py", line 39, in telemetry_middleware
raise exc
File "/venv/lib/python3.10/site-packages/karapace/api/telemetry/middleware.py", line 32, in telemetry_middleware
response: Response = await call_next(request)
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 185, in call
with collapse_excgroups():
File "/usr/local/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/venv/lib/python3.10/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 187, in call
response = await self.dispatch_func(request, call_next)
File "/venv/lib/python3.10/site-packages/karapace/api/middlewares/init.py", line 77, in set_content_types
response = await call_next(request)
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 163, in call_next
raise app_exc
File "/venv/lib/python3.10/site-packages/starlette/middleware/base.py", line 149, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/venv/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/venv/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/venv/lib/python3.10/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/venv/lib/python3.10/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/venv/lib/python3.10/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "/venv/lib/python3.10/site-packages/dependency_injector/wiring.py", line 994, in _patched
return await _async_inject(
File "src/dependency_injector/_cwiring.pyx", line 66, in _async_inject
File "/venv/lib/python3.10/site-packages/karapace/api/routers/config.py", line 62, in config_put
return await forward_client.forward_request_remote(
File "/venv/lib/python3.10/site-packages/karapace/api/forward_client.py", line 105, in forward_request_remote
return response_type.parse_raw(body) # type: ignore[return-value]
File "/venv/lib/python3.10/site-packages/pydantic/main.py", line 1277, in parse_raw
return cls.model_validate(obj)
File "/venv/lib/python3.10/site-packages/pydantic/main.py", line 627, in model_validate
return cls.pydantic_validator.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for CompatibilityResponse
compatibility
Field required [type=missing, input_value={'error_code': 42203, 'me...e, and full_transitive'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.10/v/missing