Skip to content

test(ai): live media-input integration tests per provider, runnable from GitHub Actions#1381

Open
ling-senpeng13 wants to merge 9 commits into
mainfrom
test/llm-media-integration
Open

test(ai): live media-input integration tests per provider, runnable from GitHub Actions#1381
ling-senpeng13 wants to merge 9 commits into
mainfrom
test/llm-media-integration

Conversation

@ling-senpeng13

@ling-senpeng13 ling-senpeng13 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

What

Live-API regression coverage for the provider media-input fixes (#1238 Anthropic, #1241 Gemini, #1243 Grok/Perplexity, #1246 Cohere) plus live locks for the pre-existing OpenAI-family media path — runnable from GitHub using the org-level provider secrets.

  • testChatCompletionWithImageMedia added to the key-gated live nests of AnthropicTest, GrokTest, PerplexityAITest, OpenAITest, AzureOpenAITest; new ApiKeyIntegrationTests nest in GeminiVertexTest (AI-Studio mode, GEMINI_API_KEY — the mode the server uses via conductor.ai.gemini.api-key); new BearerIntegrationTests nest in BedrockTest (AWS_BEARER_TOKEN_BEDROCK, matching the server's conductor.ai.bedrock.bearerToken — separate from the access-key nest because the CI AWS keys are provisioned for artifact publishing). CohereAITest already had the equivalent test from fix(ai): attach user media for Cohere (vision) requests #1246.
  • Each test sends the repo's deterministic token image (MELON7391, ai/src/test/resources/media/melon7391.png) to a vision-capable model and asserts the token is transcribed — a pass proves the adapter actually attached the media; the token cannot be answered by luck (pattern from test(e2e): Python media-input suite (image → vision model) agentspan-ai/agentspan#301).
  • New workflow llm-media-integration.yml: weekly cron + workflow_dispatch + PR runs when the workflow file changes. Publishes a per-provider JUnit report ("LLM Media Integration Report").

Results (this PR's workflow run — live APIs, org secrets)

Provider Model Result
Anthropic claude-haiku-4-5 ✅ pass
OpenAI gpt-4o-mini ✅ pass
Azure OpenAI gpt-4o-mini deployment ✅ pass
Gemini gemini-2.5-flash (AI Studio) ✅ pass
Grok grok-4.5 ✅ pass
Perplexity sonar ✅ pass
Bedrock us.anthropic.claude-haiku-4-5-20251001-v1:0 (bearer auth) ✅ pass
Cohere command-a-vision-07-2025 ⏭ skipped — no COHERE_API_KEY org secret; activates automatically once added

8 tests run, 7 passed, 1 skipped, 0 failed.

Fixes folded in along the way

  • Grok env-var mismatch: the org secret and tests use GROK_API_KEY, but the server bound only XAI_API_KEY. application.properties now falls back ${XAI_API_KEY:${GROK_API_KEY:}}, so either name configures the provider (XAI wins when both are set).
  • Bedrock model-id lessons (discovered by the live runs, encoded as comments): bare model ids are rejected for on-demand throughput — inference-profile ids (us. prefix) are required; and claude-3-haiku is provider-Legacy and blocked after 30 days of inactivity. The pre-existing access-key-gated Bedrock chat test was updated to the current Haiku 4.5 profile id accordingly.
  • Stale Gemini model in the Vertex live test: gemini-1.5-flashgemini-2.5-flash.

Budget control — when the paid tests run (and don't)

Live provider calls cost real money, so the triggers are deliberately narrow:

  • Unrelated changes never trigger a run. The pull_request trigger is path-filtered to the workflow file itself and ai/src/main/java/org/conductoross/conductor/ai/providers/** — the exact code these tests exist to guard. A PR touching anything else in the repo spends nothing.
  • Fork PRs structurally cannot spend: GitHub withholds org/repo secrets from fork-originated runs, so every key-gated test reports SKIPPED.
  • Rapid pushes don't stack runs: a concurrency group (llm-media-${{ github.ref }}, cancel-in-progress: true) cancels superseded runs.
  • Per-run cost is bounded: fixed 8 tests, maxTokens(100), one ~3.5 KB image — roughly 8 calls × ~300 tokens, i.e. a fraction of a cent per run.
  • Scheduled spend: weekly cron (30 5 * * 1 UTC) — pennies per year; plus on-demand workflow_dispatch.

If stricter control is ever wanted, the next steps are a PR label gate (run only with a run-llm-tests label) or moving the secrets behind a protected GitHub environment with required reviewers, which also gates cron and manual dispatches behind human approval.

Not covered — no org-level key configured ✘

These providers have no secret at the org level, so their live coverage cannot run in CI at all:

  • MISTRAL_API_KEYMistralAITest.IntegrationTests exists but can never run in CI until a key is added.
  • COHERE_API_KEY — the Cohere live tests (adapter integration test from fix(ai): attach user media for Cohere (vision) requests #1246 + the row in this suite) stay dormant; they activate automatically, zero code changes, the moment the secret is added.
  • HUGGINGFACE_API_KEYHuggingFaceTest's key-gated live nest likewise can never run in CI.
  • STABILITY_API_KEY — no key, so no live Stability coverage is possible in CI (image-generation provider; not part of the media-input fix series).

Follow-ups (not in this PR)

  • Add COHERE_API_KEY as an org secret (org admin) to activate the Cohere row — no code change needed.
  • Optionally provision MISTRAL_API_KEY / HUGGINGFACE_API_KEY / STABILITY_API_KEY the same way; the existing key-gated nests pick them up automatically.

Adds testChatCompletionWithImageMedia to the key-gated IntegrationTests
nests of Anthropic (#1238), Grok and Perplexity (#1243), and a new
GEMINI_API_KEY-gated ApiKeyIntegrationTests nest for Gemini (#1241,
AI-Studio mode — the mode the server uses with conductor.ai.gemini.api-key),
complementing the existing Cohere test (#1246). Each sends the repo's
token image (MELON7391) to a vision model and asserts transcription —
a pass proves the adapter attached the media; the token cannot be guessed.

Vision models verified against provider docs: claude-haiku-4-5,
gemini-2.5-flash, grok-4.5 (docs.x.ai image understanding), sonar
(Perplexity image attachments, data-URI image_url).

New workflow llm-media-integration.yml runs the suite on GitHub with the
org-level provider secrets (weekly cron + workflow_dispatch + PR runs on
workflow changes). Tests self-skip per absent key (COHERE_API_KEY is not
configured today), so the run stays green with per-provider status.

Verified locally: Anthropic passes against the live API; keyless
providers skip. Full gemini/grok/perplexity verification happens on the
PR's own workflow run via org secrets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ling-senpeng13 ling-senpeng13 self-assigned this Jul 22, 2026
ling-senpeng13 and others added 4 commits July 22, 2026 13:18
- OpenAITest / AzureOpenAITest: testChatCompletionWithImageMedia on
  gpt-4o-mini — both ride OpenAIResponsesChatModel's input_image path
  (pre-existing support; this locks it live at each endpoint).
- BedrockTest: new BearerIntegrationTests nest gated on
  AWS_BEARER_TOKEN_BEDROCK (the dedicated org secret, matching the
  server's conductor.ai.bedrock.bearerToken binding) — kept separate
  from the access-key nest because the CI AWS keys are provisioned for
  artifact publishing and may lack Bedrock permissions. Media flows via
  Spring AI Converse image blocks; model claude-3-haiku (vision-capable,
  same id as the existing bedrock tests).
- Workflow wires the new secrets (OPENAI, AZURE_OPENAI key+endpoint,
  AWS_BEARER_TOKEN_BEDROCK, AWS_REGION).

Verified locally: Anthropic and OpenAI pass against the live APIs;
keyless providers skip. Azure/Bedrock get their first live run on this
PR's workflow run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bearer-token auth works, but Bedrock rejects the bare model id for
on-demand throughput ("Retry your request with the ID or ARN of an
inference profile") — use the us. cross-region profile id.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… media test

claude-3-haiku is marked Legacy by the provider and Bedrock rejects it
after 30 days of inactivity ("Access denied. This Model is marked by
provider as Legacy"). Use the active Haiku 4.5 US inference profile.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- conductor.ai.grok.api-key now falls back XAI_API_KEY -> GROK_API_KEY,
  so the org secret (GROK_API_KEY, which the tests already gate on) also
  configures the server-side provider; XAI_API_KEY still wins when set.
- BedrockTest.IntegrationTests chat test: legacy bare claude-3-haiku id
  -> current us. Haiku 4.5 inference profile (same two failures the
  media test hit live: profile required, Legacy model blocked).
- GeminiVertexTest Vertex chat test: retired gemini-1.5-flash ->
  gemini-2.5-flash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ling-senpeng13
ling-senpeng13 requested a review from v1r3n July 22, 2026 20:51
…guard

PRs touching ai/.../providers/** now run the live media suite (the exact
changes these tests exist to catch), while everything else in the repo
still never triggers a paid run. Concurrency group cancels superseded
runs on rapid pushes so spend cannot stack.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ling-senpeng13
ling-senpeng13 marked this pull request as ready for review July 22, 2026 22:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant