Handling capability mismatches between models.dev and provider reality

[neilberkman]

Handling capability mismatches between models.dev and provider reality

Problem

models.dev capabilities are binary (true/false/null) and don't capture provider-specific quirks:

• Partial support (tools work, but not in streaming mode)
• API restrictions (provider limitation, not model limitation)
• Unreliable behavior (capability exists but doesn't work consistently)

When capabilities don't match reality, tests fail with no good way to handle it.

Example: Bedrock Llama 3.3 70B

models.dev says:

{ "id": "meta.llama3-3-70b-instruct-v1:0", "tool_call": true }

Reality: 19 tests, 3 failures

1. Streaming + tools: HTTP 400 "This model doesn't support tool use in streaming mode"

   • Bedrock API limitation, not model limitation

2. Type coercion: Schema wants pos_integer, model returns "30" (string)

   • Structured output works but ignores type constraints

3. Unreliable tools: With tool_choice forcing tool use, got 0 tool calls

   • Capability exists but doesn't work consistently

Current workaround: Removed from test matrix (config/config.exs, test/support/model_matrix.ex)

Possible Solutions

1. Provider capability overrides

# In amazon_bedrock provider module
def adjust_capabilities("meta.llama" <> _, caps) do
  Map.put(caps, :streaming_tools, false)
end

Pro: Accurate. Con: Duplicates models.dev data.

2. Extended models.dev schema

{
  "tool_call": { "supported": true, "streaming": false }
}

Pro: Canonical source. Con: Requires upstream changes.

3. Conditional test generation

unless has_known_issue?(provider, model, capability) do
  test "..." do
end

Pro: Precise control. Con: Messy macro logic.

Recommendation

Short term: Provider capability overrides (option 1)
Long term: Work with models.dev on richer capability metadata (option 2)

[neilberkman]

Update: Proposed Long-term Fix Upstream

I've opened an issue and PR on models.dev to address this at the source:

• Issue: Feature Request: Expand tool_call to an object for granular capabilities sst/models.dev#342
• PR: feat: Allow tool_call to be an object for granular capabilities sst/models.dev#343

Proposal

Change tool_call from a boolean to a union type that accepts either:

• true/false (backward compatible)
• Object with granular fields:
  • supported: boolean
  • streaming: boolean (optional)
  • coerces_types: boolean (optional)

This would allow models.dev to document provider-specific quirks like:

[tool_call]
supported = true
streaming = false       # Bedrock limitation
coerces_types = true    # Bedrock integer coercion

Timeline

No guarantee this will be accepted. Even if accepted, it's a breaking change for consumers, so adoption may take time.

Recommendation: Still worth implementing short-term workarounds in req_llm (provider-specific overrides or conditional test logic) while we wait to see if this gets accepted upstream.

If models.dev accepts this, we can eventually remove local overrides and consume the granular data directly.

[mikehostetler]

The local catalog_allow.exs supports overrides for model capability data - leaving this open while we track the upstream issue on models.dev

[neilberkman]

The local catalog_allow.exs supports overrides for model capability data - leaving this open while we track the upstream issue on models.dev

@mikehostetler unless I'm missing something, I don't think the current override system can handle this without code changes.

The fundamental issue is that our local overrides are still constrained by the models.dev schema. Since the schema defines tool_call as a boolean, there's no way to express granular capabilities like "supports tools but not in streaming mode" - even locally.

Concrete example:

Bedrock Llama 3.3 70B supports tools in non-streaming mode but returns HTTP 400 when combining tools with streaming (Bedrock API limitation).

In the current schema, I can only set tool_call: true or tool_call: false. There's no way to express "tools work, but not with streaming" because the schema doesn't provide fields for that distinction.

Even if I tried to hack around it by overriding tool_call with an object structure locally, the code that consumes it (capability.ex:233) expects a boolean and would misinterpret it.

To actually support this use case with local overrides alone, we'd need to invent our own extended schema locally and modify all the consuming code to understand it - essentially maintaining a parallel capability system that defeats the purpose of using models.dev.

The upstream PR solves this by making the schema itself rich enough to express these real-world constraints that multiple consumer libraries are hitting.

Does that make sense, or am I missing something about how the overrides could work here?