ChatGoogleGenerativeAI does not support `ContentBlock.Multimodal.*`

[aaronjwhiteside-appdirect]

Checked other resources

• This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import { ContentBlock, HumanMessage, SystemMessage } from "@langchain/core/messages";
import dotenv from "dotenv";
import { ChatGoogleGenerativeAI } from "@langchain/google-genai";

dotenv.config({ path: "xxxxxx/.env" });

const main = async () => {
  const model = new ChatGoogleGenerativeAI({
    model: "gemini-3-pro-preview",
    apiKey: process.env.GEMINI_API_KEY!,
    thinkingConfig: {
      includeThoughts: true,
      thinkingLevel: "HIGH",
    },
  });

  const messages = [
    new SystemMessage({
      content: "You are a helpful assistant that answers questions about the world.",
    }),
    new HumanMessage({
      contentBlocks: [
        { type: "text", text: "what is this a picture of?" } satisfies ContentBlock.Text,
        {
          type: "image",
          // sample data, replace with something real
          data: "/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/wAALCAABAAEBAREA/8QAFAABAAAAAAAAAAAAAAAAAAAACf/EABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAD8AKp//2Q==",
          mimeType: "image/jpeg",
          metadata: {
            filename: "image.jpg",
          },
        } satisfies ContentBlock.Multimodal.Image & ContentBlock.Multimodal.DataRecordBase64,
        {
          type: "text",
          text: 'The user cannot see this text only you can, they have uploaded a file.\n<file id="cmh43owcq0001rag7ce3flj36" filename="image.png" mimeType="image/png" size="178333 bytes" url="https://q4mfvgwvvhsz4evm.public.blob.vercel-storage.com/orgs/org_2Wqza6cFY4qeIUxlTTqUwr5Z62I/chats/cmh43ornj0000rag7rpnv989m/files/cmh43owcq0001rag7ce3flj36.png" />',
        } satisfies ContentBlock.Text,
      ],
    }),
  ];

  const response = await model.invoke(messages);
  console.log(response);
};

void main();

Error Message and Stack Trace (if applicable)

/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:343
      throw new Error(`Unknown content type ${content.type}`);
            ^

Error: Unknown content type image
    at _convertLangChainContentToPart (/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:343:13)
    at <anonymous> (/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:406:21)
    at Array.map (<anonymous>)
    at convertMessageContentToParts (/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:406:10)
    at messages.reduce.content (/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:473:21)
    at Array.reduce (<anonymous>)
    at convertBaseMessagesToContent (/xxxxxxxx/node_modules/@langchain/google-genai/src/utils/common.ts:448:19)
    at ChatGoogleGenerativeAI._generate (/xxxxxxxx/node_modules/@langchain/google-genai/src/chat_models.ts:846:20)
    at <anonymous> (/xxxxxxxx/node_modules/@langchain/core/src/language_models/chat_models.ts:553:46)
    at Array.map (<anonymous>)

Node.js v22.22.0

Process finished with exit code 1

Description

It appears that ChatGoogleGenerativeAI does not support ContentBlock.Multimodal.Image at all.

However, the documentations lists image input as supported. https://docs.langchain.com/oss/javascript/integrations/chat/google_generative_ai

System Info

npm info langchain

langchain@1.2.18 | MIT | deps: 5 | versions: 381
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-1.2.18.tgz
.shasum: ae1aa38a1ab02bc2e2c893ad475b18bebc7a080b
.integrity: sha512-+7P9pA57Q1zvHyeDt/MVmBJ+RXVsS7FF1Lo0h6bpxFrAjLaxkBWPG8FMI376jBEtFug6wk0CtInEH9PXPxlukA==
.unpackedSize: 3.3 MB

dependencies:
@langchain/langgraph-checkpoint: ^1.0.0 langsmith: >=0.4.0 <1.0.0 zod: ^3.25.76 || ^4
@langchain/langgraph: ^1.1.2 uuid: ^10.0.0

maintainers:

• christian-bromann mail@bromann.dev
• nfcampos nuno@boringbits.io
• jacoblee93 jacoblee93@gmail.com
• andrewnguonly andrewnguonly@gmail.com
• davidduong david@duong.cz
• hntrl hunter@hntrl.io
• hwchase17 hw.chase.17@gmail.com
• basproul braceasproul@gmail.com

dist-tags:
alpha: 1.0.0-alpha.9 latest: 1.2.18 tag-for-publishing-older-releases: 0.2.20
dev: 1.2.9 next: 1.0.0-alpha.9

published 9 hours ago by GitHub Actions npm-oidc-no-reply@github.com

npm info "@langchain/google-genai"

@langchain/google-genai@2.1.15 | MIT | deps: 2 | versions: 92
Google Generative AI integration for LangChain.js
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain-google-genai/

dist
.tarball: https://registry.npmjs.org/@langchain/google-genai/-/google-genai-2.1.15.tgz
.shasum: a82ed60c702ce05a2d758208a444d6224e04e228
.integrity: sha512-Pg+mbMP1q83T/YWjF9E9uKq/D+GhnKS/xOQQuGUIM4wjGlA4BBKtjSzjVnIl66l3CUemVMCu8XYfUnDPbHmm1A==
.unpackedSize: 499.2 kB

dependencies:
@google/generative-ai: ^0.24.0 uuid: ^11.1.0

maintainers:

• hwchase17 hw.chase.17@gmail.com
• jacoblee93 jacoblee93@gmail.com
• basproul braceasproul@gmail.com
• eric_langchain eric@langchain.dev
• andrewnguonly andrewnguonly@gmail.com
• davidduong david@duong.cz
• maddyadams madelyn.adams1@gmail.com
• sam_noyes sam@langchain.dev
• andy-langchain andy@langchain.dev
• rcasup ryan.l.casupanan@gmail.com
• hntrl hunter@hntrl.io
• christian-bromann mail@bromann.dev

dist-tags:
dev: 2.1.9 latest: 2.1.15 next: 0.1.3-rc.0

published 2 days ago by GitHub Actions npm-oidc-no-reply@github.com

[aaronjwhiteside-appdirect]

This looks like a boarder issue, no ContentBlock.Multimodal.* support at all.

testing with a pdf file upload, we get the same sort of error.

   err: {
      "type": "Error",
      "message": "Unknown content type file",
      "stack":
          Error: Unknown content type file
              at _convertLangChainContentToPart (/xxxxxxxx/.next/dev/server/chunks/node_modules_@langchain_5183fe2a._.js:2828:39)
              at /xxxxxxxx/.next/dev/server/chunks/node_modules_@langchain_5183fe2a._.js:2864:87
              at Array.map (<anonymous>)
              at convertMessageContentToParts (/xxxxxxxx/.next/dev/server/chunks/node_modules_@langchain_5183fe2a._.js:2864:78)

[aaronjwhiteside-appdirect]

ContentBlock.Multimodal.Video and ContentBlock.Multimodal.Audio should also be supported.