test(ai): live media-input integration tests per provider, runnable from GitHub Actions#1381
Open
ling-senpeng13 wants to merge 9 commits into
Open
test(ai): live media-input integration tests per provider, runnable from GitHub Actions#1381ling-senpeng13 wants to merge 9 commits into
ling-senpeng13 wants to merge 9 commits into
Conversation
Adds testChatCompletionWithImageMedia to the key-gated IntegrationTests nests of Anthropic (#1238), Grok and Perplexity (#1243), and a new GEMINI_API_KEY-gated ApiKeyIntegrationTests nest for Gemini (#1241, AI-Studio mode — the mode the server uses with conductor.ai.gemini.api-key), complementing the existing Cohere test (#1246). Each sends the repo's token image (MELON7391) to a vision model and asserts transcription — a pass proves the adapter attached the media; the token cannot be guessed. Vision models verified against provider docs: claude-haiku-4-5, gemini-2.5-flash, grok-4.5 (docs.x.ai image understanding), sonar (Perplexity image attachments, data-URI image_url). New workflow llm-media-integration.yml runs the suite on GitHub with the org-level provider secrets (weekly cron + workflow_dispatch + PR runs on workflow changes). Tests self-skip per absent key (COHERE_API_KEY is not configured today), so the run stays green with per-provider status. Verified locally: Anthropic passes against the live API; keyless providers skip. Full gemini/grok/perplexity verification happens on the PR's own workflow run via org secrets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- OpenAITest / AzureOpenAITest: testChatCompletionWithImageMedia on gpt-4o-mini — both ride OpenAIResponsesChatModel's input_image path (pre-existing support; this locks it live at each endpoint). - BedrockTest: new BearerIntegrationTests nest gated on AWS_BEARER_TOKEN_BEDROCK (the dedicated org secret, matching the server's conductor.ai.bedrock.bearerToken binding) — kept separate from the access-key nest because the CI AWS keys are provisioned for artifact publishing and may lack Bedrock permissions. Media flows via Spring AI Converse image blocks; model claude-3-haiku (vision-capable, same id as the existing bedrock tests). - Workflow wires the new secrets (OPENAI, AZURE_OPENAI key+endpoint, AWS_BEARER_TOKEN_BEDROCK, AWS_REGION). Verified locally: Anthropic and OpenAI pass against the live APIs; keyless providers skip. Azure/Bedrock get their first live run on this PR's workflow run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bearer-token auth works, but Bedrock rejects the bare model id for
on-demand throughput ("Retry your request with the ID or ARN of an
inference profile") — use the us. cross-region profile id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… media test
claude-3-haiku is marked Legacy by the provider and Bedrock rejects it
after 30 days of inactivity ("Access denied. This Model is marked by
provider as Legacy"). Use the active Haiku 4.5 US inference profile.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- conductor.ai.grok.api-key now falls back XAI_API_KEY -> GROK_API_KEY, so the org secret (GROK_API_KEY, which the tests already gate on) also configures the server-side provider; XAI_API_KEY still wins when set. - BedrockTest.IntegrationTests chat test: legacy bare claude-3-haiku id -> current us. Haiku 4.5 inference profile (same two failures the media test hit live: profile required, Legacy model blocked). - GeminiVertexTest Vertex chat test: retired gemini-1.5-flash -> gemini-2.5-flash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…guard PRs touching ai/.../providers/** now run the live media suite (the exact changes these tests exist to catch), while everything else in the repo still never triggers a paid run. Concurrency group cancels superseded runs on rapid pushes so spend cannot stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ling-senpeng13
marked this pull request as ready for review
July 22, 2026 22:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Live-API regression coverage for the provider media-input fixes (#1238 Anthropic, #1241 Gemini, #1243 Grok/Perplexity, #1246 Cohere) plus live locks for the pre-existing OpenAI-family media path — runnable from GitHub using the org-level provider secrets.
testChatCompletionWithImageMediaadded to the key-gated live nests ofAnthropicTest,GrokTest,PerplexityAITest,OpenAITest,AzureOpenAITest; newApiKeyIntegrationTestsnest inGeminiVertexTest(AI-Studio mode,GEMINI_API_KEY— the mode the server uses viaconductor.ai.gemini.api-key); newBearerIntegrationTestsnest inBedrockTest(AWS_BEARER_TOKEN_BEDROCK, matching the server'sconductor.ai.bedrock.bearerToken— separate from the access-key nest because the CI AWS keys are provisioned for artifact publishing).CohereAITestalready had the equivalent test from fix(ai): attach user media for Cohere (vision) requests #1246.MELON7391,ai/src/test/resources/media/melon7391.png) to a vision-capable model and asserts the token is transcribed — a pass proves the adapter actually attached the media; the token cannot be answered by luck (pattern from test(e2e): Python media-input suite (image → vision model) agentspan-ai/agentspan#301).llm-media-integration.yml: weekly cron +workflow_dispatch+ PR runs when the workflow file changes. Publishes a per-provider JUnit report ("LLM Media Integration Report").Results (this PR's workflow run — live APIs, org secrets)
claude-haiku-4-5gpt-4o-minigpt-4o-minideploymentgemini-2.5-flash(AI Studio)grok-4.5sonarus.anthropic.claude-haiku-4-5-20251001-v1:0(bearer auth)command-a-vision-07-2025COHERE_API_KEYorg secret; activates automatically once added8 tests run, 7 passed, 1 skipped, 0 failed.
Fixes folded in along the way
GROK_API_KEY, but the server bound onlyXAI_API_KEY.application.propertiesnow falls back${XAI_API_KEY:${GROK_API_KEY:}}, so either name configures the provider (XAI wins when both are set).us.prefix) are required; andclaude-3-haikuis provider-Legacy and blocked after 30 days of inactivity. The pre-existing access-key-gated Bedrock chat test was updated to the current Haiku 4.5 profile id accordingly.gemini-1.5-flash→gemini-2.5-flash.Budget control — when the paid tests run (and don't)
Live provider calls cost real money, so the triggers are deliberately narrow:
pull_requesttrigger is path-filtered to the workflow file itself andai/src/main/java/org/conductoross/conductor/ai/providers/**— the exact code these tests exist to guard. A PR touching anything else in the repo spends nothing.concurrencygroup (llm-media-${{ github.ref }},cancel-in-progress: true) cancels superseded runs.maxTokens(100), one ~3.5 KB image — roughly 8 calls × ~300 tokens, i.e. a fraction of a cent per run.30 5 * * 1UTC) — pennies per year; plus on-demandworkflow_dispatch.If stricter control is ever wanted, the next steps are a PR label gate (run only with a
run-llm-testslabel) or moving the secrets behind a protected GitHub environment with required reviewers, which also gates cron and manual dispatches behind human approval.Not covered — no org-level key configured ✘
These providers have no secret at the org level, so their live coverage cannot run in CI at all:
MISTRAL_API_KEY—MistralAITest.IntegrationTestsexists but can never run in CI until a key is added.COHERE_API_KEY— the Cohere live tests (adapter integration test from fix(ai): attach user media for Cohere (vision) requests #1246 + the row in this suite) stay dormant; they activate automatically, zero code changes, the moment the secret is added.HUGGINGFACE_API_KEY—HuggingFaceTest's key-gated live nest likewise can never run in CI.STABILITY_API_KEY— no key, so no live Stability coverage is possible in CI (image-generation provider; not part of the media-input fix series).Follow-ups (not in this PR)
COHERE_API_KEYas an org secret (org admin) to activate the Cohere row — no code change needed.MISTRAL_API_KEY/HUGGINGFACE_API_KEY/STABILITY_API_KEYthe same way; the existing key-gated nests pick them up automatically.