`decode_base64_to_image_file` fails on CMYK images — silently drops an entire dataset (e.g. SEEDBench_IMG_KO)

[jaeminSon]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

Summary

decode_base64_to_image() normalizes only three image modes before the caller writes the file as PNG, so any dataset containing a CMYK image raises OSError: cannot write mode CMYK as PNG. Because run.py catches per-dataset exceptions and still exits 0, the affected dataset produces no prediction file, no score, and no non-zero exit code — it simply disappears from the results.

Reproduced on main @ d21c5e9, Pillow 12.3.0.

Location

vlmeval/smp/vlm.py:156-171

def decode_base64_to_image(base64_string, target_size=-1):
    image_data = base64.b64decode(base64_string)
    image = Image.open(io.BytesIO(image_data))
    if image.mode in ('RGBA', 'P', 'LA'):      # <-- incomplete
        image = image.convert('RGB')
    ...

def decode_base64_to_image_file(base64_string, image_path, target_size=-1):
    image = decode_base64_to_image(base64_string, target_size=target_size)
    ...
    image.save(image_path)                     # format inferred from .png suffix

The conversion list enumerates three modes, but the set PNG cannot encode is larger. Verified with Pillow 12.3.0:

PNG accepts	PNG rejects (OSError: cannot write mode X as PNG)
1 L LA I I;16 P RGB RGBA	PA CMYK YCbCr LAB HSV F

CMYK and YCbCr are both native JPEG modes, so either can appear in any base64-encoded dataset. PA is also rejected despite resembling a palette mode.

Reproduction

from PIL import Image
import base64, io
from vlmeval.smp.vlm import decode_base64_to_image_file

buf = io.BytesIO()
Image.new('CMYK', (32, 32)).save(buf, format='JPEG')   # valid CMYK JPEG
b64 = base64.b64encode(buf.getvalue()).decode()

decode_base64_to_image_file(b64, '/tmp/x.png')
# OSError: cannot write mode CMYK as PNG

Real-world trigger: SEEDBench_IMG_KO (NCSOFT/K-SEED) contains CMYK images. Evaluating it produces:

[dataset] SEEDBench_IMG_KO ... cannot write mode CMYK as PNG

and then the run continues to completion with exit code 0, leaving that benchmark absent from the report. The failure mode is easy to miss: status.json records an error_message, but nothing else signals that a 14k-sample benchmark was dropped.

Suggested fix

Convert whenever the mode is not PNG-encodable, rather than listing three cases:

 def decode_base64_to_image(base64_string, target_size=-1):
     image_data = base64.b64decode(base64_string)
     image = Image.open(io.BytesIO(image_data))
-    if image.mode in ('RGBA', 'P', 'LA'):
+    # PNG cannot encode PA/CMYK/YCbCr/LAB/HSV/F; CMYK and YCbCr are native
+    # JPEG modes and do occur in released datasets.
+    if image.mode not in ('1', 'L', 'LA', 'I', 'I;16', 'P', 'RGB', 'RGBA'):
         image = image.convert('RGB')
     if target_size > 0:
         image.thumbnail((target_size, target_size))
     return image

Keeping P/LA unconverted preserves current behaviour for the modes PNG supports; only the genuinely unsupported ones are coerced.

Two adjacent points, if useful:

• encode_image_to_base64(..., fmt='JPEG') has the mirror problem: JPEG cannot encode RGBA/LA/P alpha modes.
• Independently of this bug, a dataset that raises during inference currently yields a clean exit. Surfacing a non-zero exit, or a summary line naming datasets that produced no prediction, would make failures like this visible at the point they happen rather than when someone notices a gap in the results table.

Environment

• VLMEvalKit main @ d21c5e9
• Pillow 12.3.0
• Observed while evaluating a local model over SEEDBench_IMG_KO via --mode infer

How to Reproduce / 如何复现

from vlmeval.api import LMDeployAPI # run.py:653 (get_api_model_class default)
from vlmeval.dataset import build_dataset # dataset/init.py:402
from vlmeval.inference import infer_data_job # inference.py:210
from vlmeval.smp import get_pred_file_path # smp/file.py:208

WORK, MODEL, DS = 'outputs/manual/9b/T-manual', '9b', 'SEEDBench_IMG_KO'

dataset = build_dataset(DS) # resolves name -> class
model = LMDeployAPI(model=MODEL, # run.py:281-284
api_base='http://127.0.0.1:8000/v1/chat/completions',
key='sk-admin', max_tokens=2048, temperature=0.0)

result_file = get_pred_file_path(WORK, MODEL, DS) # /9b_.xlsx
infer_data_job(model, work_dir=WORK, model_name=MODEL, # writes result_file
dataset=dataset, verbose=True, api_nproc=64)

print(dataset.evaluate(result_file, model='exact_matching', # returns the metrics
nproc=4, verbose=True))

Additional Information / 补充信息

No response

[taking-lying-flat]

@jaeminSon Thank you for the detailed report and minimal reproduction! They made it straightforward to reproduce the CMYK-to-PNG failure and trace it to VLMEvalKit's image decoder.

I've submitted an upstream fix: open-compass/VLMEvalKit#1682.

The patch converts PNG-incompatible decoded modes to RGB while retaining the existing RGBA/P/LA conversion behavior and preserving supported grayscale modes, including I;16. Local regression checks passed all 21 cases, covering CMYK JPEGs, LAB/F TIFFs, existing image modes, resizing, PNG output, and invalid-image handling. The PR contains only the production-code change; the tests were run locally.

The upstream PR is awaiting review. The separate issue you identified with dataset failures not producing a non-zero process exit remains outside this patch. I haven't run the full SEEDBench_IMG_KO benchmark locally.

Thanks again for catching this and documenting the impact on evaluation results so clearly!