QA/Automation: Expand CPT tests — Document extraction in tool call results (agentic-ai)

[gbetances089]

Implementation Spec (Agent-Ready)

GOAL

Create a new CPT E2E test class that validates the document extraction feature from PR #6999. Tests must mimic real user flows with a real LLM (gpt-5-nano), asserting that documents returned by tools are correctly delivered to the model and reasoned about.

Base branch: agentic-ai-document-tool-call-results (PR #6999's branch — tests depend on that code)

---

1. Test Class Setup

Create: connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/java/io/camunda/connector/e2e/agenticai/e2e/AiAgentE2EDocumentTestIT.java

⚠️ Class name MUST end with E2ETestIT — the Maven failsafe plugin in the it-real-llm profile only runs **/*E2ETestIT.java.

@SpringBootTest(classes = AiAgentE2ETestApplication.class)
@CamundaSpringProcessTest
@ActiveProfiles("it-real-llm")
@EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")
public class AiAgentE2EDocumentTestIT {

    @Autowired private CamundaClient camundaClient;
    @Autowired private CamundaProcessTestContext processTestContext;

    @BeforeAll
    static void setUp() {
        setAssertionTimeout(Duration.ofMinutes(3));
    }

    @BeforeEach
    void mockDocumentTools() {
        // HTTP connector is DISABLED in the Docker bundle (CONNECTOR_OUTBOUND_DISABLED=io.camunda:http-json:1)
        // So HTTP tool jobs stay open → mock workers pick them up here
        processTestContext
            .mockJobWorker("io.camunda:http-json:1")
            .withHandler((jobClient, job) -> {
                // Route by element ID, return Documents for doc-related tools
            });
    }
}

---

2. BPMN & Process

Use the existing ai-agent-e2e-openai.bpmn as base. Dynamically add document-returning tools using BpmnUtil.updateInputMappings(...) or by programmatically adding script tasks to the ad-hoc subprocess.

Alternatively, create a single new BPMN file (ai-agent-e2e-document.bpmn) following document-tool-call-results.bpmn as reference. Key elements:

• An HTTP service task with storeResponse=true that downloads a file (job will be mocked)
• Script tasks inside the AI Agent AHSP that return toolCallResult as a Document or nested structure
• The AI Agent AHSP must have the same element template config as in ai-agent-e2e-openai.bpmn (OpenAI provider, gpt-5-nano)
• Deploy ai-agent-chat-user-feedback.form alongside the BPMN for user task completion

---

3. Document Delivery via Mock Workers

The HTTP connector is disabled in Docker. Mock workers intercept the jobs. To return a real Camunda Document:

// Read PDF from test resources
var pdfBytes = getClass().getResourceAsStream("/document-tool-call-results/project-launch.pdf").readAllBytes();

// Create a Camunda Document via the client API
var documentRef = camundaClient.newCreateDocumentCommand()
    .content(new ByteArrayInputStream(pdfBytes))
    .fileName("project-launch.pdf")
    .contentType("application/pdf")
    .send()
    .join();

// Return as toolCallResult
jobClient.newCompleteCommand(job)
    .variable("toolCallResult", documentRef)
    .send().join();

If camundaClient.newCreateDocumentCommand() is not available or the agent doesn't recognize the result as a Document:

• Fallback: enable the HTTP connector for these tests, serve PDFs via WireMock accessible at host.testcontainers.internal:<port>, and let the real connector download with storeResponse=true.
• Document this decision in a code comment.

---

4. PDF Fixtures

Place in src/test/resources/document-tool-call-results/:

• project-launch.pdf — unique fact: "Project Zypherion launched on March 15, 2026"
• headcount-report.pdf — unique fact: "847 employees across 12 offices"
• author-info.pdf — unique fact: "Dr. Kael Thrennix, Chief Analytics Officer"

These can be reused from PR #6999's branch or recreated as simple single-page PDFs.

---

5. Test Scenarios

#	Method Name	Prompt	Mock Returns	Judge Assertion
1	shouldReasonAboutSingleDocument	"Download the project report and tell me the project name and launch date"	1 PDF (project-launch)	responseText contains "Project Zypherion" AND "March 15, 2026"
2	shouldReasonAboutMultipleDocuments	"Search for all available reports and summarize each one"	2 PDFs (project-launch + headcount-report)	responseText mentions BOTH "Project Zypherion" AND "847 employees"
3	shouldReasonAboutNestedDocumentStructure	"Fetch the full report with all attachments and cover page, describe everything"	Nested: {summary: "Q1", attachments: [doc1, doc2], metadata: {cover: doc3}}	responseText references all 3 facts (Zypherion, 847 employees, Dr. Kael Thrennix)
4	shouldHandleExternalDocumentReference	"Get the externally referenced specification document"	{"camunda.document.type": "external", "url": "...", "name": "Spec Sheet"}	responseText mentions content from external doc
5	shouldHandleTextAndDocumentMix	"Get the current time AND download the project report"	GetDateAndTime=now(), Download=PDF	responseText includes time value AND "Project Zypherion"
6	Negative: shouldCompleteWithoutDocumentMessageWhenNoDocumentsReturned	"What is the current date and time in Berlin?"	now() (no document)	Process completes; responseText has time; NO document-related errors
7	Negative: shouldHandleBrokenDocumentReference	"Download the corrupted report"	Return a map with camunda.document.type but invalid/missing binary	Process raises incident OR agent reports inability gracefully

---

6. User Task Completion Pattern

// Wait for user task
assertThatProcessInstance(processInstance).hasActiveElements("User_Feedback");

// Find and complete
var tasks = camundaClient.newUserTaskSearchRequest()
    .filter(f -> f.processInstanceKey(processInstance.getProcessInstanceKey()))
    .send().join();
long taskKey = tasks.items().getFirst().getUserTaskKey();

camundaClient.newCompleteUserTaskCommand(taskKey)
    .variables(Map.of("userSatisfied", true))
    .send().join();

// Assert completion
assertThatProcessInstance(processInstance).isCompleted();

---

7. Judge Prompt Requirements

Judge prompts must require content correctness (facts only obtainable from the PDF):

assertThatProcessInstance(processInstance)
    .hasVariableSatisfiesJudge("agent",
        "The agent variable contains a responseText that mentions 'Project Zypherion' "
        + "and the specific date 'March 15, 2026', proving it read and reasoned about "
        + "the PDF document content provided via the tool call");

Note: <doc /> structural correctness cannot be asserted via judge (judge only sees agent.responseText). For structural assertions, use a programmatic check on the agent.context variable if it contains the conversation, or trust the unit/integration tests from PR #6999 which already cover message structure.

---

8. Cleanup

@BeforeEach
void clearDocumentStore() {
    InMemoryDocumentStore.INSTANCE.clear();
}

---

9. Verification: How to Validate This Implementation Works

After writing the tests, the agent MUST verify:

• Compile check: mvn compile -pl connectors-e2e-test/connectors-e2e-test-agentic-ai -am passes
• Unit test run (without real LLM): mvn test -pl connectors-e2e-test/connectors-e2e-test-agentic-ai — tests should be SKIPPED (not failed) due to missing OPENAI_API_KEY
• Integration test run (with key): OPENAI_API_KEY=<key> mvn verify -pl connectors-e2e-test/connectors-e2e-test-agentic-ai -Pit-real-llm — all document tests pass
• Naming check: class ends with E2ETestIT and is in the io.camunda.connector.e2e.agenticai.e2e package
• No BPMN duplication: only one new BPMN file max, or dynamic patching used
• No external network dependency: tests use mocked tools or localhost WireMock only
• Judge prompts are specific: each requires facts unique to the test PDF, not generic statements

---

Configuration

• Model: gpt-5-nano (set in application-it-real-llm.yml)
• CI job: run-ai-agent-cpt (path filter already covers connectors-e2e-test/connectors-e2e-test-agentic-ai/**)
• Gating: @EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")

---

References

Resource	Link
Parent ticket	#7349
Feature PR	#6999
CPT infra PR	#6940
Existing E2E test (pattern to follow)	AiAgentE2ETestIT.java
Document viability test (reference)	DocumentToolCallResultsIT.java
Document BPMN (reference)	document-tool-call-results.bpmn
BpmnUtil helper	BpmnUtil.java
CPT config	application-it-real-llm.yml
ADR	connectors/agentic-ai/docs/adr/004-document-handling-in-tool-call-results.md

[gbetances089]

🧠 Implementation Context Bundle

To ensure the new document-focused E2E CPT tests can be implemented with no roadblocks or intervention, reference this context bundle.

---

1. PDF Test Files

Create three small single-page PDFs with these exact contents (use Apache PDFBox or any PDF generation tool):

File	Content
project-launch.pdf	"Project Zypherion was officially launched on March 15, 2026 by the Innovation Division."
headcount-report.pdf	"As of Q1 2026, the company employs 847 people across 12 offices worldwide."
author-info.pdf	"This report was prepared by Dr. Kael Thrennix, Chief Analytics Officer."

Place them in: src/test/resources/document-tool-call-results/

---

2. User Task Form Resource

Must be deployed alongside BPMN for user task completion to work.

• File: connectors-e2e-test/connectors-e2e-test-agentic-ai/src/test/resources/ai-agent-chat-user-feedback.form
• Variables on completion: userSatisfied (boolean), followUpInput (string, optional)

---

3. Document Creation API

Preferred approach — create a real Camunda Document in the mock worker:

var pdfBytes = getClass().getResourceAsStream("/document-tool-call-results/project-launch.pdf").readAllBytes();

var documentRef = camundaClient.newCreateDocumentCommand()
    .content(new ByteArrayInputStream(pdfBytes))
    .fileName("project-launch.pdf")
    .contentType("application/pdf")
    .send()
    .join();

jobClient.newCompleteCommand(job)
    .variable("toolCallResult", documentRef)
    .send().join();

If this API is unavailable or the agent doesn't recognize the result as a Document:

• Enable the HTTP connector for document scenarios
• Serve PDFs via WireMock at host.testcontainers.internal:<port>
• Let the real connector download with storeResponse=true
• Document this decision in a code comment

---

[gbetances089]

4. Available Test Dependencies (already on classpath)

- wiremock-standalone (com.github.tomakehurst)
- testcontainers
- awaitility
- assertj
- camunda-process-test-spring
- connector-agentic-ai (production code)
- connector-http-json
- langchain4j
- spring-boot-test

---

5. agent Variable Structure at Runtime

{
  "responseText": "The project is called Zypherion...",
  "context": { /* opaque conversation/memory state - not readable by judge */ },
  "metrics": { "modelCalls": 2, "tokenUsage": { "input": 500, "output": 200 } }
}

• Judge assertions target agent and evaluate responseText.
• context is opaque — judge cannot inspect <doc /> tags inside it.

---

6. Ad-Hoc Subprocess Tool Definition Pattern

Tools inside the AHSP are discovered by the AI Agent via bpmn:documentation (tool description) and zeebe:ioMapping inputs with fromAi(...) expressions (tool parameters).

<bpmn:scriptTask id="Download_Report" name="Download Report">
  <bpmn:documentation>Downloads a PDF report and returns it as a document for analysis.</bpmn:documentation>
  <bpmn:extensionElements>
    <zeebe:script expression="=downloadedFiles[1]" resultVariable="toolCallResult" />
    <zeebe:ioMapping>
      <zeebe:input source="=fromAi(toolCall.reportName, &quot;Name of the report to download&quot;)" target="reportName" />
    </zeebe:ioMapping>
  </bpmn:extensionElements>
</bpmn:scriptTask>

Use BpmnUtil to dynamically patch BPMN at test runtime.

---

7. Docker/Network Constraints (CPT MANAGED Mode)

- Zeebe engine: runs embedded in the test JVM
- Connectors: run in Docker (connectors-bundle image)
- HTTP connector: DISABLED in Docker via CONNECTOR_OUTBOUND_DISABLED=io.camunda:http-json:1
  → HTTP tool jobs stay open for mock workers to intercept
- To reach test JVM from Docker: use host.testcontainers.internal:<port>
- Secrets (OPENAI_API_KEY): passed via connectors-secrets in application-it-real-llm.yml
- InMemoryDocumentStore: shared in-process (test JVM only, not Docker)

---

[gbetances089]

8. Full Working Test Example (Copy This Pattern)

@SpringBootTest(classes = AiAgentE2ETestApplication.class)
@CamundaSpringProcessTest
@ActiveProfiles("it-real-llm")
@EnabledIfEnvironmentVariable(named = "OPENAI_API_KEY", matches = ".+")
public class AiAgentE2EDocumentTestIT {

    private static final String BPMN_RESOURCE = "ai-agent-e2e-document.bpmn";
    private static final String FORM_RESOURCE = "ai-agent-chat-user-feedback.form";
    private static final String PROCESS_ID = "ai-agent-e2e-document";
    private static final String HTTP_JSON_JOB_TYPE = "io.camunda:http-json:1";

    @Autowired private CamundaClient camundaClient;
    @Autowired private CamundaProcessTestContext processTestContext;

    @BeforeAll
    static void setUp() {
        setAssertionTimeout(Duration.ofMinutes(3));
    }

    @BeforeEach
    void clearDocumentStore() {
        InMemoryDocumentStore.INSTANCE.clear();
    }

    @BeforeEach
    void mockDocumentTools() {
        processTestContext
            .mockJobWorker(HTTP_JSON_JOB_TYPE)
            .withHandler((jobClient, job) -> {
                var result = switch (job.getElementId()) {
                    case "Download_Report" -> createDocumentFromResource("/document-tool-call-results/project-launch.pdf");
                    default -> null;
                };
                var cmd = jobClient.newCompleteCommand(job);
                if (result != null) {
                    cmd = cmd.variable("toolCallResult", result);
                }
                cmd.send().join();
            });
    }

    @Test
    void shouldReasonAboutSingleDocument() {
        camundaClient.newDeployResourceCommand()
            .addResourceFromClasspath(BPMN_RESOURCE)
            .addResourceFromClasspath(FORM_RESOURCE)
            .send().join();

        var processInstance = camundaClient.newCreateInstanceCommand()
            .bpmnProcessId(PROCESS_ID).latestVersion()
            .variables(Map.of("inputText", "Download the project report and tell me the project name and launch date"))
            .send().join();

        assertThatProcessInstance(processInstance).hasActiveElements("User_Feedback");

        var tasks = camundaClient.newUserTaskSearchRequest()
            .filter(f -> f.processInstanceKey(processInstance.getProcessInstanceKey()))
            .send().join();
        long taskKey = tasks.items().getFirst().getUserTaskKey();

        camundaClient.newCompleteUserTaskCommand(taskKey)
            .variables(Map.of("userSatisfied", true))
            .send().join();

        assertThatProcessInstance(processInstance).isCompleted();
        assertThatProcessInstance(processInstance)
            .hasVariableSatisfiesJudge("agent",
                "The agent variable contains a responseText that mentions 'Project Zypherion' "
                + "and the specific date 'March 15, 2026', proving it read the PDF document "
                + "content provided via a tool call result");
    }
}

---

9. What NOT to Do

❌ Don't	Why
Name class anything but *E2ETestIT	Maven failsafe only runs **/*E2ETestIT.java in it-real-llm profile
Skip @EnabledIfEnvironmentVariable	Tests will fail in local dev without API key
Call external APIs (jsonplaceholder, jokeapi)	All tools must be mocked
Forget to deploy the .form resource	User task completion will fail silently
Skip setAssertionTimeout(Duration.ofMinutes(3))	LLM calls take 30-120s
Create multiple BPMN files for variants	Use BpmnUtil or a single file with multiple tools
Assume HTTP connector is active in Docker	It's disabled — jobs are for mock workers
Put tests outside io.camunda.connector.e2e.agenticai.e2e	Breaks package conventions
Assert <doc /> tags via judge	Judge only sees responseText, not internal conversation structure

---

[gbetances089]

10. Acceptance Checklist

• Class named AiAgentE2EDocumentTestIT in io.camunda.connector.e2e.agenticai.e2e
• All 3 PDFs exist in src/test/resources/document-tool-call-results/
• Mock worker produces real Camunda Documents as tool call results
• Every scenario (single, multi, nested, external, mixed, negative×2) has a test method
• Each test deploys BOTH BPMN and .form file
• Each judge prompt requires facts unique to the test PDF
• setAssertionTimeout(Duration.ofMinutes(3)) is set
• InMemoryDocumentStore.INSTANCE.clear() in @BeforeEach
• Tests skip (not fail) when OPENAI_API_KEY is absent
• Compiles: mvn compile -pl connectors-e2e-test/connectors-e2e-test-agentic-ai -am
• If mock approach doesn't work for Documents, fallback is documented in code comments

---

References

Resource	Link
Existing E2E test (pattern)	AiAgentE2ETestIT.java
Document viability test (reference)	DocumentToolCallResultsIT.java
Document BPMN (reference)	document-tool-call-results.bpmn
BpmnUtil helper	BpmnUtil.java
CPT config	application-it-real-llm.yml
ADR	connectors/agentic-ai/docs/adr/004-document-handling-in-tool-call-results.md
Feature PR	#6999
CPT infra PR	#6940