1x1 input image can crash /v1/embeddings endpoint until pod restart while health check endpoint continues to return 200

[scottt732]

System Info

We have an image processing pipeline where we're using Infinity behind KubeAI to compute CLIP embeddings for images.

curl -vv http://kubeai/openai/v1/embeddings \
  --http1.1 -H "Authorization: Bearer blah" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "https://some.image.url/image.png",
    "model": "infinity-hosted-clip-embedding-model",
    "modality": "image"
  }'

Every now and then we get into a situation where a single bad request comes along (working on trying to isolate it). From that point on, our steady stream of embeddings, health, and metrics calls ... becomes a steady stream of just health check and metric calls (both of which continue to return 200s). The model stays running but stops receiving any embedding requests from kubeai and we essentially lose 1/n of our throughput.

INFO:     10.15.5.245:57645 - "POST /v1/embeddings HTTP/1.1" 200 OK
ERROR    2025-06-26 13:23:51,860 infinity_emb ERROR: broken batch_handler.py:557
         data stream when reading image file
         Traceback (most recent call last):
           File
         "/app/infinity_emb/inference/batch_handler.py",
         line 541, in _preprocess_batch
             feat = self._model.encode_pre(items_for_pre)
           File
         "/app/infinity_emb/transformer/vision/torch_vision
         .py", line 159, in encode_pre
             preprocessed = self.processor(
           File
         "/app/.venv/lib/python3.10/site-packages/transform
         ers/models/clip/processing_clip.py", line 109, in
         __call__
             image_features = self.image_processor(images,
         return_tensors=return_tensors,
         **image_processor_kwargs)
           File
         "/app/.venv/lib/python3.10/site-packages/transform
         ers/image_processing_utils.py", line 41, in
         __call__
             return self.preprocess(images, **kwargs)
           File
         "/app/.venv/lib/python3.10/site-packages/transform
         ers/models/clip/image_processing_clip.py", line
         307, in preprocess
             images = [convert_to_rgb(image) for image in
         images]
           File
         "/app/.venv/lib/python3.10/site-packages/transform
         ers/models/clip/image_processing_clip.py", line
         307, in <listcomp>
             images = [convert_to_rgb(image) for image in
         images]
           File
         "/app/.venv/lib/python3.10/site-packages/transform
         ers/image_transforms.py", line 776, in
         convert_to_rgb
             image = image.convert("RGB")
           File
         "/app/.venv/lib/python3.10/site-packages/PIL/Image
         .py", line 995, in convert
             self.load()
           File
         "/app/.venv/lib/python3.10/site-packages/PIL/Image
         File.py", line 312, in load
             raise _get_oserror(err_code, encoder=False)
         OSError: broken data stream when reading image
         file

Any ideas?

Information

• Docker + cli
• pip + cli
• pip + usage of Python interface

Tasks

• An officially supported CLI command
• My own modifications

Reproduction

I'm trying to find the image URL that reproduces this. Will update back if I find one.

[scottt732]

I was able to find a repro that works consistently directly against infinity (updated issue to remove KubeAI).

curl -vv http://localhost:50574/v1/embeddings \
  --http1.1 -H "Authorization: Bearer doesnt-matter" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9sXsdrgAAAAASUVORK5CYII=",
    "model": "repro", 
    "modality": "image"
  }'

The issue has to do with data: encoded images that are 1x1 with 1 or 3 channels (RGB) where the normalization that happens in huggingface/transformers can't determine the channel order in [1,1,3] (channels first or channels last) b/c it's ambiguous.

The bigger problem here is that infinity doesn't handle this error gracefully.

* Host localhost:50574 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:50574...
* Connected to localhost (::1) port 50574
> POST /v1/embeddings HTTP/1.1
> Host: localhost:50574
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer doesnt-matter
> Content-Type: application/json
> Content-Length: 204
>
* upload completely sent off: 204 bytes

We never get a reply... It just eventually times out. All future requests to the /v1/embeddings endpoint also fail from that point on until we kill & replace the pod. The /health and /metrics endpoints keep returning 200s

KubeAI assumed the pod was healthy b/c of the 200's and spread traffic around to all infinity instances. And this becomes like a slow motion DoS (once each pod has encountered the bad input data once, no more CLIP embeddings--of text or images).

❯ curl -vv http://localhost:50574/v1/embeddings \
  --http1.1 -H "Authorization: Bearer doesnt-matter" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "hi",
    "model": "repro", 
    "modality": "text"
  }'
* Host localhost:50574 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:50574...
* Connected to localhost (::1) port 50574
> POST /v1/embeddings HTTP/1.1
> Host: localhost:50574
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer doesnt-matter
> Content-Type: application/json
> Content-Length: 90
>
* upload completely sent off: 90 bytes

[michaelfeil]

Which version of infinity are you using? Never releases have validation to not allow 1x1 images.

[wirthual]

Just checked, the current check is only applied to images provided via URL. The linked PR applies the check to all methods. With the check, the server returns a "Bad Request" for images smaller than 3x3 which also applies to the example provided. This will circumvent the problem described.

[scottt732]

Thanks!. Just want to make sure this didn't get lost above:
This unhandled exception broke the /v1/embeddings call that triggered it (never got a response) and prevented all future executions of that endpoint from working at all. Maybe missing a with block or try/finally on a queue slot or something?

Image is: michaelf34/infinity:0.0.76

[wirthual]

I added additional changes in this branch: #614

@scottt732 Could you give this branch a spin and report back if it solves your issue?

[michaelfeil]

I think @scottt732 was not on the most recent branch. This issue should be fixed long time ago.