Skip to content

feat(mistral-ai): update model YAMLs [bot]#1014

Merged
harshiv-26 merged 2 commits into
mainfrom
bot/update-mistral-ai-20260513-022001
May 13, 2026
Merged

feat(mistral-ai): update model YAMLs [bot]#1014
harshiv-26 merged 2 commits into
mainfrom
bot/update-mistral-ai-20260513-022001

Conversation

@models-bot
Copy link
Copy Markdown
Contributor

@models-bot models-bot Bot commented May 13, 2026

Auto-generated by poc-agent for provider mistral-ai.


Note

Low Risk
Low risk: this is a metadata-only change to a model capability YAML and does not alter runtime logic. Risk is limited to potential downstream consumers that validate or act on supported modalities.

Overview
Updates the mistral-ai model config for ministral-14b-latest to advertise support for document (doc) inputs by adding doc to modalities.input in the YAML.

Reviewed by Cursor Bugbot for commit 0506b63. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions
Copy link
Copy Markdown
Contributor

/test-models

@harshiv-26
Copy link
Copy Markdown
Collaborator

Gateway test results

  • Total: 10
  • Passed: 8
  • Failed: 0
  • Validation failed: 2
  • Errored: 0
  • Skipped: 0
  • Success rate: 80.0%
Provider Model Scenarios
mistral-ai ministral-14b-latest success: tool-call:stream, params:stream, params, parallel-tool-call, parallel-tool-call:stream, tool-call, json-output:stream, json-output

validation_failure: structured-output, structured-output:stream
Failures (2)

mistral-ai/ministral-14b-latest — structured-output (validation_failure)

Error:

Traceback (most recent call last):
  File "/tmp/tmp2ts2tn1p/snippet.py", line 44, in <module>
    raise Exception("VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)")
Exception: VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)
Code snippet
from openai import OpenAI
import json

client = OpenAI(api_key="***", base_url="https://internal.devtest.truefoundry.tech/api/llm")

response_schema = json.loads('''{
  "title": "CalendarEvent",
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "date": { "type": "string" },
    "participants": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["name", "date", "participants"],
  "additionalProperties": false
}''')

response = client.chat.completions.create(
    model="test-v2-mistral-ai/ministral-14b-latest",
    messages=[
        {"role": "system", "content": "Extract the event information as JSON."},
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hi, how can I help you"},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday. Extract the event details as JSON."},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "CalendarEvent", "schema": response_schema}},
    stream=False,
)

import json as _json

_content = response.choices[0].message.content
print(_content)

if not _content:
    raise Exception("VALIDATION FAILED: structured-output - response content is empty")

_parsed = _json.loads(_content)

if "name" not in _parsed or "date" not in _parsed or "participants" not in _parsed:
    raise Exception("VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)")

if not isinstance(_parsed.get("participants"), list):
    raise Exception("VALIDATION FAILED: structured-output - 'participants' is not a list, schema not enforced")

if set(_parsed.keys()) != {"name", "date", "participants"}:
    raise Exception(
        f"VALIDATION FAILED: structured-output - unexpected keys present: {set(_parsed.keys())}"
    )

print("VALIDATION: structured-output SUCCESS")

mistral-ai/ministral-14b-latest — structured-output:stream (validation_failure)

Error:

Traceback (most recent call last):
  File "/tmp/tmplb706x8j/snippet.py", line 49, in <module>
    raise Exception("VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)")
Exception: VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)
Code snippet
from openai import OpenAI
import json

client = OpenAI(api_key="***", base_url="https://internal.devtest.truefoundry.tech/api/llm")

response_schema = json.loads('''{
  "title": "CalendarEvent",
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "date": { "type": "string" },
    "participants": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["name", "date", "participants"],
  "additionalProperties": false
}''')

response = client.chat.completions.create(
    model="test-v2-mistral-ai/ministral-14b-latest",
    messages=[
        {"role": "system", "content": "Extract the event information as JSON."},
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hi, how can I help you"},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday. Extract the event details as JSON."},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "CalendarEvent", "schema": response_schema}},
    stream=True,
)

import json as _json

_accumulated = ""
for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content is not None:
            _accumulated += delta.content
            print(delta.content, end="", flush=True)

if not _accumulated:
    raise Exception("VALIDATION FAILED: structured-output stream - no content received")

_parsed = _json.loads(_accumulated)

if "name" not in _parsed or "date" not in _parsed or "participants" not in _parsed:
    raise Exception("VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)")

if not isinstance(_parsed.get("participants"), list):
    raise Exception("VALIDATION FAILED: structured-output stream - 'participants' is not a list, schema not enforced")

if set(_parsed.keys()) != {"name", "date", "participants"}:
    raise Exception(
        f"VALIDATION FAILED: structured-output stream - unexpected keys present: {set(_parsed.keys())}"
    )

print("\nVALIDATION: structured-output stream SUCCESS")

@harshiv-26 harshiv-26 enabled auto-merge (squash) May 13, 2026 12:11
@github-actions
Copy link
Copy Markdown
Contributor

/test-models

@harshiv-26 harshiv-26 merged commit 2a739a3 into main May 13, 2026
8 checks passed
@harshiv-26 harshiv-26 deleted the bot/update-mistral-ai-20260513-022001 branch May 13, 2026 12:12
@harshiv-26
Copy link
Copy Markdown
Collaborator

Gateway test results

  • Total: 10
  • Passed: 8
  • Failed: 0
  • Validation failed: 2
  • Errored: 0
  • Skipped: 0
  • Success rate: 80.0%
Provider Model Scenarios
mistral-ai ministral-14b-latest success: tool-call:stream, parallel-tool-call:stream, parallel-tool-call, params, tool-call, params:stream, json-output, json-output:stream

validation_failure: structured-output, structured-output:stream
Failures (2)

mistral-ai/ministral-14b-latest — structured-output (validation_failure)

Error:

Traceback (most recent call last):
  File "/tmp/tmp0p1ot45_/snippet.py", line 44, in <module>
    raise Exception("VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)")
Exception: VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)
Code snippet
from openai import OpenAI
import json

client = OpenAI(api_key="***", base_url="https://internal.devtest.truefoundry.tech/api/llm")

response_schema = json.loads('''{
  "title": "CalendarEvent",
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "date": { "type": "string" },
    "participants": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["name", "date", "participants"],
  "additionalProperties": false
}''')

response = client.chat.completions.create(
    model="test-v2-mistral-ai/ministral-14b-latest",
    messages=[
        {"role": "system", "content": "Extract the event information as JSON."},
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hi, how can I help you"},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday. Extract the event details as JSON."},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "CalendarEvent", "schema": response_schema}},
    stream=False,
)

import json as _json

_content = response.choices[0].message.content
print(_content)

if not _content:
    raise Exception("VALIDATION FAILED: structured-output - response content is empty")

_parsed = _json.loads(_content)

if "name" not in _parsed or "date" not in _parsed or "participants" not in _parsed:
    raise Exception("VALIDATION FAILED: structured-output - missing expected fields (name, date, participants)")

if not isinstance(_parsed.get("participants"), list):
    raise Exception("VALIDATION FAILED: structured-output - 'participants' is not a list, schema not enforced")

if set(_parsed.keys()) != {"name", "date", "participants"}:
    raise Exception(
        f"VALIDATION FAILED: structured-output - unexpected keys present: {set(_parsed.keys())}"
    )

print("VALIDATION: structured-output SUCCESS")

mistral-ai/ministral-14b-latest — structured-output:stream (validation_failure)

Error:

Traceback (most recent call last):
  File "/tmp/tmpisw8qox3/snippet.py", line 49, in <module>
    raise Exception("VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)")
Exception: VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)
Code snippet
from openai import OpenAI
import json

client = OpenAI(api_key="***", base_url="https://internal.devtest.truefoundry.tech/api/llm")

response_schema = json.loads('''{
  "title": "CalendarEvent",
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "date": { "type": "string" },
    "participants": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["name", "date", "participants"],
  "additionalProperties": false
}''')

response = client.chat.completions.create(
    model="test-v2-mistral-ai/ministral-14b-latest",
    messages=[
        {"role": "system", "content": "Extract the event information as JSON."},
        {"role": "user", "content": "Hi"},
        {"role": "assistant", "content": "Hi, how can I help you"},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday. Extract the event details as JSON."},
    ],
    response_format={"type": "json_schema", "json_schema": {"name": "CalendarEvent", "schema": response_schema}},
    stream=True,
)

import json as _json

_accumulated = ""
for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content is not None:
            _accumulated += delta.content
            print(delta.content, end="", flush=True)

if not _accumulated:
    raise Exception("VALIDATION FAILED: structured-output stream - no content received")

_parsed = _json.loads(_accumulated)

if "name" not in _parsed or "date" not in _parsed or "participants" not in _parsed:
    raise Exception("VALIDATION FAILED: structured-output stream - missing expected fields (name, date, participants)")

if not isinstance(_parsed.get("participants"), list):
    raise Exception("VALIDATION FAILED: structured-output stream - 'participants' is not a list, schema not enforced")

if set(_parsed.keys()) != {"name", "date", "participants"}:
    raise Exception(
        f"VALIDATION FAILED: structured-output stream - unexpected keys present: {set(_parsed.keys())}"
    )

print("\nVALIDATION: structured-output stream SUCCESS")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant