Skip to content

Heavy Tests

Heavy Tests #971

Manually triggered January 22, 2026 16:24
Status Failure
Total duration 11m 16s
Artifacts 13

heavy-tests.yml

on: workflow_dispatch
Matrix: integration-tests
Fit to window
Zoom out
Zoom in

Annotations

25 errors
capabilities-tests
Process completed with exit code 1.
AIAgentIntegrationTest.[11] model=LLModel(provider=OpenRouter, id=qwen/qwen-2.5-72b-instruct, capabilities=[Temperature, Speculation, Tools, Completion, Standard, ToolChoice], contextLength=131072, maxOutputTokens=8192)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=OpenRouter, id=qwen/qwen-2.5-72b-instruct, capabilities=[Temperature, Speculation, Tools, Completion, Standard, ToolChoice], contextLength=131***72, maxOutputTokens=8192) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[10] model=LLModel(provider=OpenRouter, id=deepseek/deepseek-chat-v3-0324, capabilities=[Temperature, Speculation, Tools, Completion], contextLength=163800, maxOutputTokens=163800)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=OpenRouter, id=deepseek/deepseek-chat-v3-***324, capabilities=[Temperature, Speculation, Tools, Completion], contextLength=1638***, maxOutputTokens=1638***) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[9] model=LLModel(provider=Google, id=gemini-2.5-flash, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1048576, maxOutputTokens=65536)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=Google, id=gemini-2.5-flash, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1***48576, maxOutputTokens=65536) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[8] model=LLModel(provider=Google, id=gemini-2.5-pro, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1048576, maxOutputTokens=65536)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=Google, id=gemini-2.5-pro, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1***48576, maxOutputTokens=65536) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[6] model=LLModel(provider=Anthropic, id=claude-sonnet-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=200000, maxOutputTokens=64000)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=Anthropic, id=claude-sonnet-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=2***, maxOutputTokens=64***) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[5] model=LLModel(provider=Anthropic, id=claude-haiku-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=200000, maxOutputTokens=64000)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=Anthropic, id=claude-haiku-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=2***, maxOutputTokens=64***) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[4] model=LLModel(provider=Anthropic, id=claude-opus-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=200000, maxOutputTokens=64000)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=Anthropic, id=claude-opus-4-5, capabilities=[Temperature, Tools, ToolChoice, Image, Document, Completion], contextLength=2***, maxOutputTokens=64***) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[3] model=LLModel(provider=OpenAI, id=gpt-5.1-codex, capabilities=[Completion, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, MultipleChoices, Responses], contextLength=400000, maxOutputTokens=128000)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=OpenAI, id=gpt-5.1-codex, capabilities=[Completion, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, MultipleChoices, Responses], contextLength=4***, maxOutputTokens=128***) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[2] model=LLModel(provider=OpenAI, id=gpt-4.1, capabilities=[Temperature, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, Completion, MultipleChoices, Completions, Responses], contextLength=1047576, maxOutputTokens=32768)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=OpenAI, id=gpt-4.1, capabilities=[Temperature, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, Completion, MultipleChoices, Completions, Responses], contextLength=1***47576, maxOutputTokens=32768) expected:<calculator> but was:<["calculator"]>
AIAgentIntegrationTest.[1] model=LLModel(provider=OpenAI, id=gpt-5.2, capabilities=[Completion, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, MultipleChoices, Completions, Responses], contextLength=400000, maxOutputTokens=128000)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/agent/AIAgentIntegrationTest.kt#L318
org.opentest4j.AssertionFailedError: The calculator tool was not called for model LLModel(provider=OpenAI, id=gpt-5.2, capabilities=[Completion, Basic, Standard, Speculation, Tools, ToolChoice, Image, Document, MultipleChoices, Completions, Responses], contextLength=4***, maxOutputTokens=128***) expected:<calculator> but was:<["calculator"]>
agent-tests
Process completed with exit code 1.
MultipleLLMPromptExecutorIntegrationTest.[7] model=LLModel(provider=Google, id=gemini-3-pro-preview, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1048576, maxOutputTokens=65536)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/executor/MultipleLLMPromptExecutorIntegrationTest.kt#L245
ai.koog.prompt.executor.clients.LLMClientException: Error from client: GoogleLLMClient Error from client: GoogleLLMClient Status code: 429 Error body: { "error": { "code": 429, "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n* Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model_per_day, limit: ***", "status": "RESOURCE_EXHAUSTED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Learn more about Gemini API quotas", "url": "https://ai.google.dev/gemini-api/docs/rate-limits" } ] }, { "@type": "type.googleapis.com/google.rpc.QuotaFailure", "violations": [ { "quotaMetric": "generativelanguage.googleapis.com/generate_requests_per_model_per_day", "quotaId": "GenerateRequestsPerDayPerProjectPerModel" } ] } ] } }
multiple-llm-executor-tests
Process completed with exit code 1.
SingleLLMPromptExecutorIntegrationTest.[7] model=LLModel(provider=Google, id=gemini-3-pro-preview, capabilities=[Temperature, Completion, MultipleChoices, Image, Video, Audio, Tools, ToolChoice, Basic, Standard], contextLength=1048576, maxOutputTokens=65536)[jvm]: integration-tests/src/jvmTest/kotlin/ai/koog/integration/tests/executor/SingleLLMPromptExecutorIntegrationTest.kt#L292
ai.koog.prompt.executor.clients.LLMClientException: Error from client: GoogleLLMClient Error from client: GoogleLLMClient Status code: 429 Error body: { "error": { "code": 429, "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, head to: https://ai.google.dev/gemini-api/docs/rate-limits. To monitor your current usage, head to: https://ai.dev/rate-limit. \n* Quota exceeded for metric: generativelanguage.googleapis.com/generate_requests_per_model_per_day, limit: ***", "status": "RESOURCE_EXHAUSTED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Learn more about Gemini API quotas", "url": "https://ai.google.dev/gemini-api/docs/rate-limits" } ] }, { "@type": "type.googleapis.com/google.rpc.QuotaFailure", "violations": [ { "quotaMetric": "generativelanguage.googleapis.com/generate_requests_per_model_per_day", "quotaId": "GenerateRequestsPerDayPerProjectPerModel" } ] } ] } }
single-llm-executor-tests
Process completed with exit code 1.

Artifacts

Produced during runtime
Name Size Digest
ai-analysis-agent-tests-971 Expired
181 Bytes
sha256:1ed4f152db21657f9f92e9f5b3452a60620acf3aab1d06bda7ab464c52e66d6e
ai-analysis-capabilities-tests-971 Expired
181 Bytes
sha256:638a7d7840836f9b0fef1c1d845603516e864bf9ef977edbd0005859034697a2
ai-analysis-multiple-llm-executor-tests-971 Expired
181 Bytes
sha256:de060c4f715691fb955e5a687e64ad6cd77770c69a66a806c8a26cff7682858a
ai-analysis-single-llm-executor-tests-971 Expired
181 Bytes
sha256:04feef38082b905b0772bc89d967d8fe22bcfa1dc9c9fe5c4c9d4a018c55fba8
reports-ubuntu-latest-agent-tests
476 KB
sha256:65730e56ed5123fc5458cd519f1f4f0683f8586570372b085efc9a10b88e7807
reports-ubuntu-latest-anthropic-schema-test
292 KB
sha256:d35bc9de7e5240b8a9fccb944d91990312ba78c74d4bb135cbe61c574a3543d7
reports-ubuntu-latest-bedrock-credentials-test
288 KB
sha256:10e40dee0617dae63deb06ba99699598c89d589e239a915c1974bf32a1a5a2a2
reports-ubuntu-latest-bedrock-tests
288 KB
sha256:aa03ee73b297bb2e8095a860b57b05fc1d182e2af31351b0bfdd515862c32978
reports-ubuntu-latest-capabilities-tests
316 KB
sha256:bdf0c4653de9377eb23f76f9423b352e3fe2c1d49597b8c624de10f790f2fb35
reports-ubuntu-latest-embeddings-test
288 KB
sha256:7270ff485f45493db467f7b2f485c4ab2d069fa3cb1059f1a0c370d1bd8d0b1e
reports-ubuntu-latest-multiple-llm-executor-tests
520 KB
sha256:d366bd78c2a46df94b57c3e4519e94850808f91d6a11597463928687938a4ca0
reports-ubuntu-latest-other-executor-tests
288 KB
sha256:6584b892e59e0d3ddba8da20853d476f53745d817f45c74f1aa39f715d58c5e5
reports-ubuntu-latest-single-llm-executor-tests
523 KB
sha256:8a50c989a3460ac57cebf5a0d7a5f28596509ae5a8272480f4e2edfbb7654f8a