Skip to content

QA: Document handling in tool call results (agentic-ai) — test plan and tracking #7349

@gbetances089

Description

@gbetances089

Summary

This QA ticket tracks the validation effort for the new document extraction feature introduced in PR #6999, which moves documents from agent tool call results into synthetic user messages as explicit DocumentContent blocks. Coverage includes both automation via CPT (Camunda Process Test) and manual cross-provider verification.

Scope

  • Validate that documents provided by BPMN/HTTP/MCP tools are correctly delivered to the LLM via synthetic UserMessages, visible as content blocks identifiable via tags.
  • Assert: LLM can actually reason about PDF/image/text documents, not just see metadata.
  • Regression check: no missed or duplicated documents, no fallback to base64-in-text, and existing non-document tools/functions remain unaffected.

Acceptance Criteria

  • Automated tests (CPT E2E, see subticket) cover single/multi/nested/external document scenarios
  • Manual E2E verification across major cloud providers (OpenAI, Anthropic, Bedrock, etc.; see subticket)
  • PDF/image/text documents are all delivered and reasoned about in LLM responses
  • Known gaps (tool call mocking, file serving, CI triggers) are documented or worked around

Subtasks

  • [Automation] Expand CPT E2E test suite to cover document scenarios (see below)
  • [Manual] Conduct cross-provider and conversation audit tests (see below)

Related:

  • Implementation: PR #6999
  • CPT infra: PR #6940
  • ADR: connectors/agentic-ai/docs/adr/004-document-handling-in-tool-call-results.md

Metadata

Metadata

Assignees

Labels

agentic-aicomponent:qaTask containing all details related to QAqa:requiredWill trigger the QA workflow

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions