[BUG]: TogetherAI, SambaNova and Z.ai drop chat-history images; four providers send the internal `attachments` field

[marmar9615-cloud]

How are you running AnythingLLM?

All versions

What happened?

Workspace LLM set to TogetherAI with a vision model (meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo).

1. Turn 1: attach an image in the prompt box and ask "What is in this picture?". The model answers about the image.
2. Turn 2: ask "What colour is it?" without attaching anything.

On turn 2 the model answers as if no image were attached.

I expected the image to still be part of the history on turn 2. That is what happens on OpenAI and the other providers that got this in #2919 ("Adds support for vision models to be able to see images in the chat history beyond the initial image chat prompt message"). SambaNova and Z.ai behave like TogetherAI.

Where it is in the code (reviewed at master eb7df1e8):

• convertToPromptHistory (server/utils/helpers/chat/responses.js:283) stores each earlier user turn as { role, content, attachments }. It adds the attachments property at line 330.
• formatChatHistory (responses.js:370-403) rebuilds that message with the provider's own #generateContent, so the image becomes a content part again. 24 providers call it, for example server/utils/AiProviders/openAi/index.js:122.
• These four spread the raw history instead:
  • server/utils/AiProviders/togetherAi/index.js:173: ...chatHistory,
  • server/utils/AiProviders/sambanova/index.js:108
  • server/utils/AiProviders/zai/index.js:104
  • server/utils/AiProviders/cerebras/index.js:178

Two effects follow:

1. The history user message keeps content as a plain string, so the provider does not send the image again.
2. The provider spreads the internal attachments property (file name, mime type and the full base64 data URL) into the message unchanged. The pinned openai client (4.95.1) forwards it. With the client pointed at a local server standing in for the provider, the message that arrived had the keys role,content,attachments. I have not checked with a live key whether any of the four APIs rejects or ignores that field.

Cerebras differs. Its #generateContent is a text-only stub (cerebras/index.js:140-163, "There are not cerebras models that support vision"), so Cerebras never sends the image, not even on turn 1. The only defect there is the stray attachments field.

groq and ppio leave this out on purpose. groq/index.js:94 says historical attachments are omitted, and ppio/index.js:139 declares attachments unsupported.

TogetherAI has had vision models since #2666 (v1.3.0) and was not in the file list of #2919 (v1.4.0), so it has been missing this since v1.4.0. Z.ai (#4573, v1.9.1), SambaNova (#4943, v1.11.0) and Cerebras (#5699, v1.14.0) came later with the same ...chatHistory shape. All four are still like this in v1.16.1 and on master at eb7df1e8.

Are there known steps to reproduce?

In the product, run the two turns above on a TogetherAI, SambaNova or Z.ai workspace with a vision model.

Without a key, build each provider with a stub client and call constructPrompt with a history turn that carries an attachment, the shape convertToPromptHistory produces. Each header's yes/no comes from the provider file on disk. The history user message per provider at eb7df1e8:

### openAi (control)  (provider file calls formatChatHistory: yes)
history user message: image content part = true; attachments key present = false
{"role":"user","content":[{"type":"input_text","text":"What is in this picture?"},{"type":"input_image","image_url":"data:image/png;base64,AAAA"}]}

### togetherAi  (provider file calls formatChatHistory: no)
history user message: image content part = false; attachments key present = true
{"role":"user","content":"What is in this picture?","attachments":[{"name":"cat.png","mime":"image/png","contentString":"data:image/png;base64,AAAA"}]}

### sambanova  (provider file calls formatChatHistory: no)
history user message: image content part = false; attachments key present = true
{"role":"user","content":"What is in this picture?","attachments":[{"name":"cat.png","mime":"image/png","contentString":"data:image/png;base64,AAAA"}]}

### zai  (provider file calls formatChatHistory: no)
history user message: image content part = false; attachments key present = true
{"role":"user","content":"What is in this picture?","attachments":[{"name":"cat.png","mime":"image/png","contentString":"data:image/png;base64,AAAA"}]}

### cerebras  (provider file calls formatChatHistory: no)
history user message: image content part = false; attachments key present = true
{"role":"user","content":"What is in this picture?","attachments":[{"name":"cat.png","mime":"image/png","contentString":"data:image/png;base64,AAAA"}]}

The fix is the same one line the other 24 providers have: ...formatChatHistory(chatHistory, this.#generateContent),. I have a PR ready, with a test.

LLM Provider & Model (if applicable)

TogetherAI / meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo. Same on SambaNova and Z.ai with a vision model. Cerebras for the stray field only.

Embedder Provider & Model (if applicable)

Any. Not involved.