Skip to content

test: Add CPT E2E tests for document extraction in tool call results#7353

Draft
Copilot wants to merge 3 commits into
mainfrom
copilot/add-cpt-e2e-document-extraction-tests
Draft

test: Add CPT E2E tests for document extraction in tool call results#7353
Copilot wants to merge 3 commits into
mainfrom
copilot/add-cpt-e2e-document-extraction-tests

Conversation

Copilot AI commented May 26, 2026

Copy link
Copy Markdown
Contributor

Adds end-to-end CPT tests validating that the AI Agent connector correctly delivers PDF documents returned by mock tool workers to the LLM, and that the LLM can reason about their content.

New files

  • ai-agent-e2e-document.bpmn — Process with 5 HTTP document tools (Download_Report, Search_Documents, Fetch_Report, Get_External_Document, Download_Corrupted_Report) + GetDateAndTime script task
  • document-tool-call-results/*.pdf — Three PDF fixtures with distinct, queryable content (Project Zypherion / 847 employees / Dr. Kael Thrennix)
  • AiAgentE2EDocumentTestIT.java — 7 test scenarios covering the document delivery surface

Test scenarios

# Scenario Assertion
1 Single CamundaDocumentReferenceModel from tool result LLM quotes project name + date from PDF
2 List of documents from tool result LLM summarises both PDFs
3 Documents in nested map (attachments list + metadata.cover) LLM references all three document facts
4 InlineDocumentReferenceModel (no Zeebe fetch needed) LLM reports inline text content
5 Mixed: text tool + document tool in same turn LLM uses both tool results
6 No document tools used Process completes normally, no document errors
7 Broken document reference (non-existent ID) hasActiveIncidents() — connector exhausts retries

Document delivery mechanism

Mock workers store PDFs in embedded Zeebe via newCreateDocumentCommand(), then return the reference wrapped in CamundaDocumentReferenceModel — which serialises with the "camunda.document.type": "camunda" discriminator required by ObjectDeserializer. The Docker connector fetches content from embedded Zeebe via gRPC.

var response = camundaClient.newCreateDocumentCommand()
    .content(new ByteArrayInputStream(pdfBytes))
    .fileName(fileName).contentType("application/pdf")
    .send().join();

return new CamundaDocumentReferenceModel(
    response.getStoreId(), response.getDocumentId(), response.getContentHash(),
    new CamundaDocumentMetadataModel("application/pdf", null, null, fileName, null, null, null));

Tests are gated on OPENAI_API_KEY and run under the it-real-llm Spring profile, matching the existing AiAgentE2ETestIT pattern.

Copilot AI requested a review from gbetances089 May 26, 2026 12:57
Copilot AI changed the title [WIP] Add CPT E2E tests for document extraction in tool call results Add CPT E2E tests for document extraction in tool call results May 26, 2026
@gbetances089 gbetances089 force-pushed the copilot/add-cpt-e2e-document-extraction-tests branch from 95958c3 to 920b6e3 Compare May 27, 2026 07:01
@gbetances089 gbetances089 changed the title Add CPT E2E tests for document extraction in tool call results test: Add CPT E2E tests for document extraction in tool call results May 27, 2026
@gbetances089 gbetances089 requested a review from Copilot May 27, 2026 16:31
Copilot AI and others added 2 commits May 27, 2026 18:32
Co-authored-by: gbetances089 <2312478+gbetances089@users.noreply.github.com>
Replaces getClass().getResourceAsStream() with
AiAgentE2EDocumentTestIT.class.getResourceAsStream() in createDocumentRef()
to address CodeQL security finding (unsafe use of getResource for
potentially extendable classes).
@gbetances089 gbetances089 force-pushed the copilot/add-cpt-e2e-document-extraction-tests branch from 104739c to 92819dc Compare May 27, 2026 16:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new CPT end-to-end test suite that exercises the AI Agent connector’s ability to ingest and reason over documents returned via tool call results (including single docs, lists, nested structures, inline docs, and failure cases).

Changes:

  • Added a dedicated BPMN process (ai-agent-e2e-document) exposing multiple “document tools” (HTTP JSON tasks) plus a time tool.
  • Added a new real-LLM gated integration test (AiAgentE2EDocumentTestIT) covering 7 document-delivery scenarios, including a broken-reference incident case.
  • Added PDF fixtures under document-tool-call-results/ used by the mock tool workers as document content.

Reviewed changes

Copilot reviewed 2 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/ai-agent-e2e-document.bpmn New E2E process definition that models document-returning tools for the AI Agent connector to call.
connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/java/io/camunda/connector/e2e/agenticai/e2e/AiAgentE2EDocumentTestIT.java New CPT E2E ITs validating document extraction/propagation from tool call results and LLM reasoning over PDFs.
connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/project-launch.pdf PDF fixture used to validate single-document and nested-document reasoning.
connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/headcount-report.pdf PDF fixture used to validate multi-document reasoning (e.g., headcount fact extraction).
connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/document-tool-call-results/author-info.pdf PDF fixture used to validate nested metadata/cover document reasoning.

Comment on lines +401 to +405
var pdfBytes =
Objects.requireNonNull(
AiAgentE2EDocumentTestIT.class.getResourceAsStream(resourcePath),
"PDF resource not found: " + resourcePath)
.readAllBytes();
Comment on lines +420 to +421
new CamundaDocumentMetadataModel(
"application/pdf", null, null, fileName, null, null, null));
- Wrap getResourceAsStream().readAllBytes() in try-with-resources to
  avoid leaking file handles across repeated test runs
- Replace hard-coded CamundaDocumentMetadataModel constructor args with
  new CamundaDocumentMetadataModel(new DocumentMetadataImpl(response.getMetadata()))
  so the stored metadata (size, contentType, etc.) is reflected correctly

Addresses review comments PRRT_kwDOIGZYus6FKe84 and PRRT_kwDOIGZYus6FKe9j
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

QA/Automation: Expand CPT tests — Document extraction in tool call results (agentic-ai)

4 participants