ChatVertexAI does not support ContentBlock.Multimodal.* #9958
Description
Checked other resources
- This is a bug, not a usage question. For questions, please use the LangChain Forum (https://forum.langchain.com/).
- I added a very descriptive title to this issue.
- I searched the LangChain.js documentation with the integrated search.
- I used the GitHub search to find a similar question and didn't find it.
- I am sure that this is a bug in LangChain.js rather than my code.
- The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
Example Code
import { ContentBlock, HumanMessage, SystemMessage } from "@langchain/core/messages";
import dotenv from "dotenv";
import { ChatVertexAI } from "@langchain/google-vertexai";
import { logger } from "@/src/utils/Logger";
dotenv.config({ path: "xxxxxxx/.env" });
const main = async () => {
// /tmp/gcloud.json exists and is valid
const model = new ChatVertexAI({
model: "gemini-2.5-pro",
reasoningEffort: "high",
});
const messages = [
new SystemMessage({
content: "You are a helpful assistant that answers questions about the world.",
}),
new HumanMessage({
contentBlocks: [
{ type: "text", text: "what is this a picture of?" } satisfies ContentBlock.Text,
{
type: "image",
// sample data, replace with something real
data: "/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/wAALCAABAAEBAREA/8QAFAABAAAAAAAAAAAAAAAAAAAACf/EABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAD8AKp//2Q==",
mimeType: "image/jpeg",
metadata: {
filename: "image.jpg",
},
} satisfies ContentBlock.Multimodal.Image & ContentBlock.Multimodal.DataRecordBase64,
{
type: "text",
text: 'The user cannot see this text only you can, they have uploaded a file.\n<file id="cmh43owcq0001rag7ce3flj36" filename="image.jpg" mimeType="image/jpeg" size="7833 bytes" />',
} satisfies ContentBlock.Text,
],
}),
];
const response = await model.invoke(messages);
console.log(response);
};
void main();Error Message and Stack Trace (if applicable)
/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:628
throw new Error(
^
Error: Unsupported type "image" received while converting message to message parts: {"type":"image","data":"/9j/4AAQSkZJRgABAQEASABIAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/wAALCAABAAEBAREA/8QAFAABAAAAAAAAAAAAAAAAAAAACf/EABQQAQAAAAAAAAAAAAAAAAAAAAD/2gAIAQEAAD8AKp//2Q==","mimeType":"image/jpeg","metadata":{"filename":"image.jpg"}}
at messageContentComplexToPart (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:628:15)
at <anonymous> (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:647:11)
at Array.map (<anonymous>)
at messageContentComplexToParts (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:644:30)
at messageContentToParts (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:667:28)
at roleMessageToContent (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:718:46)
at baseMessageToContent (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:827:14)
at <anonymous> (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:1621:7)
at Array.map (<anonymous>)
at formatBaseMessageContents (/xxxxxxxx/node_modules/@langchain/google-common/src/utils/gemini.ts:1620:61)
Node.js v22.22.0
Description
ChatVertexAI does not support any ContentBlock.Multimodal.Image or ContentBlock.Multimodal.File content blocks, in either DataRecordBase64 or DataRecordUrl varieties.
DataRecordUrl should be supported to allow for passing in files Stored either Google Cloud Storage or youtube links for video uploads.
System Info
npm info "@langchain/google-vertexai"
@langchain/google-vertexai@2.1.15 | MIT | deps: 1 | versions: 80
LangChain.js support for Google Vertex AI
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain-google-vertexai/
dist
.tarball: https://registry.npmjs.org/@langchain/google-vertexai/-/google-vertexai-2.1.15.tgz
.shasum: 86f4e314748d2b28259b35e58b31bdcecaf53c2e
.integrity: sha512-p3klocNvePVnSytM9bcbnHIfUjOK42nQ4wvgdMOCI2FsUsHLg4He7pVGqGsIMfh6krKmiPcN8yadkomK+zh3NQ==
.unpackedSize: 90.8 kB
dependencies:
@langchain/google-gauth: 2.1.15
maintainers:
- hwchase17 hw.chase.17@gmail.com
- jacoblee93 jacoblee93@gmail.com
- basproul braceasproul@gmail.com
- eric_langchain eric@langchain.dev
- andrewnguonly andrewnguonly@gmail.com
- davidduong david@duong.cz
- maddyadams madelyn.adams1@gmail.com
- sam_noyes sam@langchain.dev
- andy-langchain andy@langchain.dev
- rcasup ryan.l.casupanan@gmail.com
- hntrl hunter@hntrl.io
- christian-bromann mail@bromann.dev
dist-tags:
dev: 2.1.9 latest: 2.1.15
published 5 days ago by GitHub Actions npm-oidc-no-reply@github.com
npm info langchain
langchain@1.2.18 | MIT | deps: 5 | versions: 381
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/libs/langchain/
keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores
dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-1.2.18.tgz
.shasum: ae1aa38a1ab02bc2e2c893ad475b18bebc7a080b
.integrity: sha512-+7P9pA57Q1zvHyeDt/MVmBJ+RXVsS7FF1Lo0h6bpxFrAjLaxkBWPG8FMI376jBEtFug6wk0CtInEH9PXPxlukA==
.unpackedSize: 3.3 MB
dependencies:
@langchain/langgraph-checkpoint: ^1.0.0 langsmith: >=0.4.0 <1.0.0 zod: ^3.25.76 || ^4
@langchain/langgraph: ^1.1.2 uuid: ^10.0.0
maintainers:
- christian-bromann mail@bromann.dev
- nfcampos nuno@boringbits.io
- jacoblee93 jacoblee93@gmail.com
- andrewnguonly andrewnguonly@gmail.com
- davidduong david@duong.cz
- hntrl hunter@hntrl.io
- hwchase17 hw.chase.17@gmail.com
- basproul braceasproul@gmail.com
dist-tags:
alpha: 1.0.0-alpha.9 latest: 1.2.18 tag-for-publishing-older-releases: 0.2.20
dev: 1.2.9 next: 1.0.0-alpha.9
published 3 days ago by GitHub Actions npm-oidc-no-reply@github.com