test: Add CPT E2E tests for document extraction in tool call results#7353
Draft
Copilot wants to merge 3 commits into
Draft
test: Add CPT E2E tests for document extraction in tool call results#7353Copilot wants to merge 3 commits into
Copilot wants to merge 3 commits into
Conversation
7 tasks
Copilot
AI
changed the title
[WIP] Add CPT E2E tests for document extraction in tool call results
Add CPT E2E tests for document extraction in tool call results
May 26, 2026
95958c3 to
920b6e3
Compare
Co-authored-by: gbetances089 <2312478+gbetances089@users.noreply.github.com>
Replaces getClass().getResourceAsStream() with AiAgentE2EDocumentTestIT.class.getResourceAsStream() in createDocumentRef() to address CodeQL security finding (unsafe use of getResource for potentially extendable classes).
104739c to
92819dc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new CPT end-to-end test suite that exercises the AI Agent connector’s ability to ingest and reason over documents returned via tool call results (including single docs, lists, nested structures, inline docs, and failure cases).
Changes:
- Added a dedicated BPMN process (
ai-agent-e2e-document) exposing multiple “document tools” (HTTP JSON tasks) plus a time tool. - Added a new real-LLM gated integration test (
AiAgentE2EDocumentTestIT) covering 7 document-delivery scenarios, including a broken-reference incident case. - Added PDF fixtures under
document-tool-call-results/used by the mock tool workers as document content.
Reviewed changes
Copilot reviewed 2 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/ai-agent-e2e-document.bpmn | New E2E process definition that models document-returning tools for the AI Agent connector to call. |
| connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/java/io/camunda/connector/e2e/agenticai/e2e/AiAgentE2EDocumentTestIT.java | New CPT E2E ITs validating document extraction/propagation from tool call results and LLM reasoning over PDFs. |
| connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/project-launch.pdf | PDF fixture used to validate single-document and nested-document reasoning. |
| connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/headcount-report.pdf | PDF fixture used to validate multi-document reasoning (e.g., headcount fact extraction). |
| connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/author-info.pdf | PDF fixture used to validate nested metadata/cover document reasoning. |
Comment on lines
+401
to
+405
| var pdfBytes = | ||
| Objects.requireNonNull( | ||
| AiAgentE2EDocumentTestIT.class.getResourceAsStream(resourcePath), | ||
| "PDF resource not found: " + resourcePath) | ||
| .readAllBytes(); |
Comment on lines
+420
to
+421
| new CamundaDocumentMetadataModel( | ||
| "application/pdf", null, null, fileName, null, null, null)); |
- Wrap getResourceAsStream().readAllBytes() in try-with-resources to avoid leaking file handles across repeated test runs - Replace hard-coded CamundaDocumentMetadataModel constructor args with new CamundaDocumentMetadataModel(new DocumentMetadataImpl(response.getMetadata())) so the stored metadata (size, contentType, etc.) is reflected correctly Addresses review comments PRRT_kwDOIGZYus6FKe84 and PRRT_kwDOIGZYus6FKe9j
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds end-to-end CPT tests validating that the AI Agent connector correctly delivers PDF documents returned by mock tool workers to the LLM, and that the LLM can reason about their content.
New files
ai-agent-e2e-document.bpmn— Process with 5 HTTP document tools (Download_Report,Search_Documents,Fetch_Report,Get_External_Document,Download_Corrupted_Report) +GetDateAndTimescript taskdocument-tool-call-results/*.pdf— Three PDF fixtures with distinct, queryable content (Project Zypherion/847 employees/Dr. Kael Thrennix)AiAgentE2EDocumentTestIT.java— 7 test scenarios covering the document delivery surfaceTest scenarios
CamundaDocumentReferenceModelfrom tool resultattachmentslist +metadata.cover)InlineDocumentReferenceModel(no Zeebe fetch needed)hasActiveIncidents()— connector exhausts retriesDocument delivery mechanism
Mock workers store PDFs in embedded Zeebe via
newCreateDocumentCommand(), then return the reference wrapped inCamundaDocumentReferenceModel— which serialises with the"camunda.document.type": "camunda"discriminator required byObjectDeserializer. The Docker connector fetches content from embedded Zeebe via gRPC.Tests are gated on
OPENAI_API_KEYand run under theit-real-llmSpring profile, matching the existingAiAgentE2ETestITpattern.