Triage agent should validate that file paths referenced in issues exist in the repo

[fullsend-ai-retro]

What happened

Issue #1580 in konflux-ci/integration-service specified that the file gitops/snapshot_integration_test_status.go should be modified to remove jsonschema usage. This file does not exist in the repository. The correct file is pkg/integrationteststatus/integration_test_status.go. The code agent correctly identified the right file and made the change, but the human reviewer flagged this discrepancy on Jun 10: 'issue states that gitops/snapshot_integration_test_status.go file should have remove jsonschema, but this file was not modified in this changes at all.' The review agent also noted this as an informational finding: 'Issue #1580 lists target file as gitops/snapshot_integration_test_status.go which does not exist in the repo.'

What could go better

The triage agent (or whichever agent authored issue #1580) should validate file paths before including them in the issue body. A simple ls or find on the referenced paths would have caught that gitops/snapshot_integration_test_status.go doesn't exist. The code agent recovered from this by searching for jsonschema usage itself, but this wastes tokens and risks the agent modifying wrong files if partial matches exist. I'm moderately confident this is a recurring pattern — agents generating issues based on codebase understanding without validating specific file references. My uncertainty is around whether the issue was human-authored or agent-authored; if human-authored, the triage agent should still validate paths during its analysis phase.

Proposed change

Add a file path validation step to the triage agent. When the triage agent processes or refines an issue that references specific file paths, it should verify each path exists in the repository using ls or stat. If a path doesn't exist, the agent should: (1) search for likely alternatives (e.g., grep for the filename or key terms), (2) update the issue body with corrected paths, and (3) note the correction. This validation should also apply to the code agent at the start of its run — before beginning implementation, it should verify all file paths mentioned in the issue and flag any that don't exist. This could be implemented as an early step in the triage and code agent definitions.

Validation criteria

Over the next 10 issues processed by the triage agent that reference specific file paths: (1) all referenced paths should exist in the repository, (2) if a path doesn't exist, the agent should note the correction in the issue, and (3) no downstream code agent should encounter a non-existent file path from an issue without prior warning.

---

Generated by retro agent from konflux-ci/integration-service#1583

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 1:36 PM UTC · Completed 1:38 PM UTC
Commit: 4ed6da4 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Category: Feature · Severity: Medium

Problem

The triage agent does not verify whether file paths referenced in issues actually exist in the target repository. This was surfaced by a real incident where an issue referenced gitops/snapshot_integration_test_status.go (which didn't exist) instead of the correct pkg/integrationteststatus/integration_test_status.go. The code agent recovered by searching independently, but this wastes tokens and risks wrong-file modifications.

Recommended Approach

Add a file path validation step to the triage agent's analysis phase:

1. Detect path-like strings in the issue body (strings with slashes and file extensions)
2. Verify each path exists using ls or stat
3. If a path doesn't exist, search for likely alternatives by filename or key terms
4. Note corrections in the triage output and update the issue body

As defense-in-depth, also consider adding an early validation step in the code agent definition (related to but distinct from #2008, which covers broader acceptance criteria validation).

Proposed Test Case

Scenario: Triage agent detects non-existent file path
  Given an issue referencing "src/nonexistent/file.go"
  When the triage agent processes the issue
  Then it detects the invalid path
  And searches for alternative paths matching the filename
  And notes the correction in its output

Scenario: Triage agent passes through valid file paths
  Given an issue referencing a file that exists in the repo
  When the triage agent processes the issue
  Then no path correction is flagged

Scenario: Issues without file paths are unaffected
  Given an issue with no file path references
  When the triage agent processes the issue
  Then the validation step is a no-op

Validation Criteria (from issue)

Over the next 10 issues with file path references: all paths should exist, corrections should be noted, and no downstream code agent should encounter unwarned non-existent paths.

---

Labels: Retro-originated feature request targeting the triage agent's analysis phase, with medium priority given the token waste and review friction impact.

[fullsend-ai-prioritize]

RICE Priority Score: 1.2

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	This affects all repositories processed by fullsend where issues reference specific file paths. The retro originated from a real customer repo (konflux-ci/integration-service), and the issue notes this is likely a recurring pattern across agent-generated issues. Multiple customers using the triage agent would benefit.
Impact	1	Medium impact — the code agent can recover by searching independently, so it's not blocking. However, invalid paths waste tokens, risk wrong-file modifications, and cause review friction (as seen in the PR #1583 review comment). The improvement is noticeable but has an existing workaround.
Confidence	0.8	High confidence — the issue is well-described with a concrete real-world incident, clear proposed solution, specific test cases, and validation criteria. The scope is well-bounded to the triage agent's analysis phase.
Effort	1	Medium effort — requires detecting path-like strings in issue bodies, verifying them against the repo, searching for alternatives, and updating the issue. Touches the triage agent definition and potentially the code agent as defense-in-depth. Well-scoped but involves regex/heuristic work for path detection.

Formula: (1.5 x 1 x 0.8) / 1 = 1.2