Fireworks forwards unsupported file content blocks

[mdrxy]

Package

langchain-fireworks

Summary

langchain-fireworks currently translates canonical LangChain file content blocks into OpenAI-style Chat Completions {"type": "file", ...} blocks and sends them to Fireworks. Fireworks Chat Completions rejects these payloads because it only supports text, image_url, and video_url content chunks.

This shows up when an agent/tool returns a PDF or other binary file as a canonical content block, for example:

{"type": "file", "base64": "...", "mime_type": "application/pdf"}

langchain-fireworks routes it through convert_to_openai_data_block(..., api="chat/completions"), which produces:

{
    "type": "file",
    "file": {
        "file_data": "data:application/pdf;base64,...",
        "filename": "..."
    },
}

Fireworks then rejects the request with:

Unexpected content chunk type file. We only support 'text', 'image_url', 'video_url'

Reproduction

from langchain_core.messages import HumanMessage
from langchain_fireworks.chat_models import _convert_message_to_dict

msg = HumanMessage(
    content=[
        {
            "type": "file",
            "base64": "JVBERi0=",
            "mime_type": "application/pdf",
            "filename": "x.pdf",
        }
    ]
)

print(_convert_message_to_dict(msg))

Current output includes an unsupported Fireworks payload:

{
    "role": "user",
    "content": [
        {
            "type": "file",
            "file": {
                "file_data": "data:application/pdf;base64,JVBERi0=",
                "filename": "x.pdf",
            },
        }
    ],
}

Expected behavior

langchain-fireworks should not send unsupported file blocks to Fireworks Chat Completions.

For canonical file blocks, it should raise a clear local error before making the API call, for example:

Fireworks Chat Completions does not support file/PDF content blocks. Convert the file to text first, or use a provider/model that supports file inputs.

Image and video blocks should continue to be translated to Fireworks-supported image_url / video_url blocks.

Actual behavior

The unsupported {"type": "file", ...} payload is forwarded to Fireworks and the remote API rejects it.

Root cause

langchain-fireworks currently treats all canonical data content blocks the same:

if is_data_content_block(block):
    formatted.append(
        convert_to_openai_data_block(block, api="chat/completions")
    )
    continue

This was added in #37090 to translate multimodal content blocks for Fireworks. It works for images because OpenAI-style image blocks become image_url, which Fireworks supports. But it also allows file blocks, and OpenAI's Chat Completions file content part is not supported by Fireworks.

There is also a unit test that currently asserts this unsupported translation:

def test_format_message_content_translates_v1_file_block_base64() -> None:
    ...
    assert formatted == [
        {
            "type": "file",
            "file": {
                "file_data": "data:application/pdf;base64,JVBERi0=",
                "filename": "x.pdf",
            },
        },
    ]

That test should likely be changed to expect a clear ValueError for Fireworks.

Suggested fix

Make Fireworks' content formatter provider-specific:

• Continue translating image blocks to image_url.
• Continue translating supported video blocks to video_url.
• Reject file blocks locally with a clear error.
• Add/adjust tests so PDF/file blocks do not produce a Fireworks wire payload.

Longer-term, callers that want to use PDFs/documents with Fireworks should extract text first and send the extracted text as a text block.

[keenborder786]

Mason Daugherty (@mdrxy) Hello, is this issue still available to work on? I’d be happy to implement a fix for it.