[Feature]: Evaluator: pluggable model providers + drop hardcoded `temperature` support OpenAI GPT‑5.6 Sol and Anthropic Claude Opus 4.7 / 4.8

[msalheb]

Description

The evaluation framework (scripts/aidlc-evaluator) is hard‑wired to the Amazon Bedrock Converse API on the bedrock-runtime endpoint, and its scorer sends a hardcoded temperature. This makes it impossible to evaluate against several current Bedrock models by config alone. Please add a thin model‑provider abstraction and make temperature optional.

Current limitations

Execution agents are Converse‑only. packages/execution/src/aidlc_runner/agents/executor.py (~L220) and simulator.py (~L137) build strands.models.bedrock.BedrockModel(model_id=...), which calls Converse on bedrock-runtime. OpenAI GPT‑5.6 Sol (openai.gpt-5.6-sol) is served only on the bedrock-mantle endpoint via the Responses API (.../openai/v1/responses), so setting model_id: openai.gpt-5.6-sol fails at invoke time.
Scorer hardcodes temperature and is Converse‑only. packages/qualitative/src/qualitative/scorer.py (~L217) calls client.converse(..., inferenceConfig={"maxTokens": ..., "temperature": 0.0}). Anthropic Claude Opus 4.7 and 4.8 reject temperature (ValidationException: temperature is deprecated for this model), so the scorer cannot run on claude-opus-4-7 or claude-opus-4-8 — the evaluator is effectively stuck on opus-4-6. Opus 4.7+ replaced temperature with an effort control (high/medium/low, effort-2025-11-24 beta header).
Config assumes global/Converse conventions. config/default.yaml uses ids like global.anthropic.claude-opus-4-6-v1; GPT‑5.6 Sol is in‑region only (e.g. us‑east‑1), with no geo/global inference profile.

Proposed change

Honor the existing models.<role>.provider field:
    provider: "bedrock" → current BedrockModel (Converse) path (unchanged default).
    provider: "openai_responses" (or auto‑detect model_id starting with openai.) → build strands.models.openai_responses.OpenAIResponsesModel with bedrock_mantle_config={"region": <region>, "boto_session": <session>}. Strands (already a dependency, ≥1.44.0) mints Bedrock bearer tokens from the standard AWS credential chain, so no separate API key is required.
Scorer: route through the same abstraction (or the OpenAI Responses SDK against the Mantle base URL) instead of a raw client.converse call; make temperature optional/omittable (don't send it for Opus 4.7/4.8), and optionally support an effort setting for Opus 4.7+.

Implementation notes / gotchas

Mantle base‑URL path: GPT‑5.6 Sol uses https://bedrock-mantle.{region}.api.aws/openai/v1, which differs from the generic .../v1 template in Strands' _openai_bedrock.py; the provider must allow a base_url (path) override.
Extras: requires the strands-agents[openai] extra (aws_bedrock_token_generator); add to pyproject.toml.
Region: support in‑region‑only models (skip the global./geo prefix assumption).

Acceptance criteria

run.py full
/
batch
succeed with
model_id: openai.gpt-5.6-sol
,
provider: openai_responses
, region
us-east-1
, using the standard AWS credential chain.
Scorer (and executor/simulator) run on
claude-opus-4-7 and claude-opus-4-8
with
no temperature sent
, and on GPT‑5.6 Sol.
effort
(
high
/
medium
/
low
) is optionally configurable per role for Opus 4.7+ (ignored where unsupported).
Existing Bedrock/Converse runs (Anthropic ≤4.6) are unchanged by default.
Config presets added under
config/
(e.g.
config/gpt-5-6-sol.yaml
,
config/opus-4-8.yaml
) and documented in
README.md
.
Unit tests cover provider selection (bedrock vs openai_responses) and the scorer's optional‑temperature / effort behavior.

Use Case

We want to benchmark the AI‑DLC workflow across the newest frontier models — including the latest Anthropic Claude (Opus 4.7 / 4.8) and non‑Anthropic models (OpenAI GPT‑5.6 Sol) — to compare quality, token cost, and cross‑model consistency, which is a core purpose of the evaluator (NFR + qualitative scoring across models).

Today "just change the model id" does not work for these models:

Opus 4.7 / 4.8 fail immediately in the scorer because of the hardcoded temperature=0.0, forcing the evaluator to stay on the older opus-4-6.
GPT‑5.6 Sol can't be reached at all because it's only on bedrock-mantle/Responses, not Converse/bedrock-runtime.

This blocks anyone trying to validate rule/prompt changes against current‑generation models or produce an up‑to‑date cross‑model comparison report. A small provider abstraction + optional temperature removes the blocker and future‑proofs the evaluator as more models ship on the Mantle/Responses path.

Version

v1 (current)

Area

Other

Additional Context

GPT‑5.6 Sol model card (endpoints/APIs, in‑region only, Responses‑only): https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-sol.html

API compatibility matrix by model: https://docs.aws.amazon.com/bedrock/latest/userguide/models-api-compatibility.html

Bedrock OpenAI / Mantle inference docs: https://docs.aws.amazon.com/bedrock/latest/userguide/inference-openai.html

Why not just wait for Converse support on GPT‑5.6 Sol? The split is architectural, not a temporary gap: open‑weight OpenAI models (gpt-oss-*) are on bedrock-runtime and ARE Converse‑compatible, but the proprietary GPT‑5.x line (5.4 → 5.6 Sol/Terra/Luna) is served only on bedrock-mantle via the Responses API. AWS designed Mantle to expose these models through their native OpenAI surface rather than Converse, so a Converse path for Sol is not expected — supporting the Responses provider is required, not optional.

Strands already ships the needed provider (strands/models/openai_responses.py, strands/models/_openai_bedrock.py), so this is primarily wiring + config, not new transport code.