Configurable CI failure analysis agent with feedback and escalation #1746

Summary

Provide a reusable Fullsend agent that investigates CI test failures using configured evidence sources, records evidence-based findings, learns from confirmed outcomes, and supports human-approved escalation to a configured forge or issue tracker. A deployment should be able to adopt the agent without changing its code for a particular CI system, organization, product, or issue project.

This is a generic upstream enhancement based on the use case in fullsend-ai/agents#1735. The configuration and any source-specific inspection skills belong to the deployment or reusable integrations.

Proposed behavior

1. A configured trigger supplies a failure report or issue containing available test, job, build, version, platform, and error metadata, plus references to logs or artifacts.
2. The agent uses configured retrieval skills and credentials to fetch current failure evidence from logs, external services, or structured stores. It records which sources were checked and which could not be accessed.
3. The agent analyzes the current evidence, compares it with relevant confirmed historical cases, and publishes a structured result to the configured destination.
4. A maintainer can confirm or correct the result. The confirmed outcome becomes available to future investigations.
5. A maintainer can explicitly approve escalation of a confirmed defect to a configured forge or issue tracker. The agent checks for an existing matching issue before creating one and links the records in both directions where supported.

Analysis result

The result should contain:

• Status: analyzed, partial evidence, or needs information.
• Failure classification, such as product defect, test or automation defect, infrastructure issue, or transient failure.
• Root-cause hypothesis, affected component, and calibrated confidence.
• Specific supporting evidence with source references and relevant excerpts.
• Sources attempted, retrieval failures, and limitations affecting confidence.
• Recommended next steps and related historical cases, when available.
• Provenance: which findings come from current evidence, which come from historical matches, and which were confirmed by a maintainer.

Use analyzed when current evidence supports a conclusion, partial evidence when current evidence supports only a limited conclusion, and needs information when there is too little current evidence to assess the failure. A single inaccessible source does not automatically invalidate other sufficient evidence, but its absence must be disclosed. The agent must not present a historical match or cached result as a fresh analysis of the current failure.

Configuration and integration boundaries

• The deployment selects the trigger, input mapping, evidence retrieval skills, credentials, output destination, knowledge store, and escalation target.
• The common input identifies the failure, its test and job context, available error details, and evidence references. Each retrieval skill returns accessible evidence with source references or a specific access error.
• Source-specific parsing and authentication live in the configured retrieval integration. The agent consumes a common failure context and evidence format.
• External evidence sources may include an observability service such as Datadog or a key-value database. A deployment configures the connection, credentials, namespace, and mapping from failure identifiers to lookup keys. The agent reads only the records needed for the reported failure and records their source and timestamp.
• A key-value database can also supply historical cases through the configured knowledge-store interface. Reading existing records and writing maintainer feedback are separate capabilities; feedback must be persisted by an explicitly configured writable store.
• A deployment can omit integrations it does not use. The upstream agent does not require a particular CI provider, log host, dashboard, product, organization, or issue tracker.
• Credentials and private log content must follow the deployment's access controls. Published evidence should avoid exposing secrets or private data to a broader audience.
• For Jira escalation, CI_ANALYSIS_ESCALATION_JIRA_PROJECT selects the destination project when Jira is configured. No project key is hardcoded upstream.

Feedback and learning

Store the failure signature, relevant context, analysis, evidence references, related cases, and final maintainer-confirmed outcome in a searchable knowledge store. A maintainer correction must supersede the original hypothesis for that case and be retrievable for future failures.

For a similar failure, the agent should compare the current evidence with prior cases, cite the matches, and explain any difference in classification. Retrieval and confirmed feedback are sufficient for the initial learning mechanism; model retraining is not required.

Human-approved escalation

Classification alone must not create an external issue. A human must explicitly confirm the defect and request escalation through a configured approval action; the decision and approver must be recorded. Before creating an issue, the agent should search the configured forge or tracker for a match using the failure signature and relevant context, and surface an existing issue when found. A new issue should include the confirmed analysis, evidence references, failure metadata, recommended action, and a link to the source report. The source report should link back to the created issue where the destination supports it.

Acceptance criteria

• The same agent configuration model works with at least two distinct evidence-source integrations without changes to the agent's analysis logic.
• A configured key-value source can look up failure evidence or historical cases by mapped identifiers, and its records are cited with source and timestamp.
• A failure report with accessible current logs yields a structured result with classification, confidence, component, cited evidence, provenance, and next steps.
• Missing or inaccessible evidence yields a useful partial or needs-information result that identifies the access problem and the information needed to continue.
• Historical matches are identified as historical and do not replace inspection of current evidence.
• Maintainer corrections are stored and influence later analyses of similar failures.
• No external issue is created without explicit human confirmation; duplicate checks and cross-links are applied on escalation.
• No CI provider, log source, organization, product, or escalation project is hardcoded in the upstream agent.