Dragon-AI-Agent PR Analysis: Mondo Repository

[dragon-ai-agent]

Dragon-AI-Agent PR Analysis: Mondo Repository

This is an analysis of success rate of agentic actions in this repo. This analysis can be used to tune instructions and verifiers.

Overview

Date: 2025-12-20
Repository: monarch-initiative/mondo
Agent Account: dragon-ai-agent

Statistics

Metric	Count	Percentage
Total PRs	26	100%
Merged	11	42%
Closed (unmerged)	14	54%
Open	1	4%

Success Rate: 11/25 completed PRs = 44%

First-Try Success Rate

Of the merged PRs (from extracted data with category info):

Category	Count	Percentage
Merged without modifications	4	26.7%
Merged with modifications	11	73.3%
Total merged (in dataset)	15	100%

First-try success rate: 4/15 = 26.7%

This is critically low - nearly 3 out of 4 merged PRs required modifications. The agent struggles to get submissions right on the first attempt.

Merged PRs (Success)

PR	Title	Merged Date
#9726	Add viral respiratory tract infection as parent to common cold	2025-12-01
#9724	Add logical definition to renal infectious disease	2025-11-03
#9717	Add infectious disease as parent to renal infectious disease	2025-11-03
#9652	Add skin basal cell carcinoma as parent	2025-10-14
#9409	Add new term avoidant/restrictive food intake disorder	2025-11-03
#9363	Refactor Addison's disease classification	2025-08-07
#9283	Add HP:0003002 xref to breast carcinoma	2025-07-03
#9261	Fix CDC URLs	2025-10-10
#9209	Add SubClassOf relationship for immunodeficiency 120	2025-06-13
#9198	Add cutaneous solitary mastocytoma as subclass	2025-06-13
#8843	Update synonyms of FATWO	2025-05-28

Failed PRs Checklist

Each PR below was closed without merging and requires analysis:

• #9734 - Simplify ochronosis disorder - Work already done by another contributor
• #9482 - Add argyrophilic grain disease - Duplicate attempt Consider additional sources for chromosomal rearrangement diseases #7
• #9456 - Add argyrophilic grain disease - Duplicate attempt OWLEntityRenamer ignores some URIs, leaving ghost alt_ids #6
• #9455 - Add argyrophilic grain disease - Duplicate attempt Validate low confidence IDs #5
• #9427 - Add argyrophilic grain disease - Duplicate attempt test new version in OLS #4
• #9376 - Add argyrophilic grain disease - Duplicate attempt sparql query to generate equivalence axioms not working #3
• #9351 - Update argyrophilic grain disease annotations - Duplicate attempt Rename #2, failed to follow instructions
• #9311 - Add argyrophilic grain disease - Initial attempt, wrong annotation format
• #9330 - Fix Lynch syndrome subtypes - Duplicate of existing work
• #9199 - Make immunodeficiency 120 a subclass - Destructive edit; created new PR instead of updating
• #9173 - Obsolete cone-rod dystrophy 12 - Replacement PR, still not merged
• #9172 - Obsolete cone-rod dystrophy 12 - Wrong obsoletion reason, replaced by Obsolete cone-rod dystrophy 12 and merge with PROM1-related retinopathy #9173
• #9167 - Add equine spinocerebellar ataxia - OWL structure error (Class and Individual)
• #8868 - Add CLAUDE.md - Duplicate of Adding instructions for AI #9019

Open PRs (Pending)

a

• #9725 - Add tuberculous pneumothorax as child of tuberculosis

Failure Mode Summary

Failure Mode	Count	PRs
Created new PR instead of updating existing	9	#9482, #9456, #9455, #9427, #9376, #9351, #9173, #9172, #9199
Wrong annotation/reference format	7	#9311, #9351, #9456, #9455, #9427, #9376, #9482
Duplicate of existing work	3	#9734, #9330, #8868
Destructive edit (removed instead of added)	2	#9734, #9199
Technical OWL/OBO structure error	1	#9167
Failed to follow reviewer instructions precisely	5	#9311, #9351, #9199, #9172, #9173

Critical Issues Identified

1. The Argyrophilic Grain Disease Saga (7 PRs for 1 term!)

This single term request (#9279/#9426) resulted in 7 separate PRs over 2 months:

• Add argyrophilic grain disease (MONDO:7770001) #9311 → Update argyrophilic grain disease term annotations #9351 → Add new term for argyrophilic grain disease (MONDO:7770001) #9376 → Add MONDO:7770002 argyrophilic grain disease #9427 → Add MONDO:7770002 argyrophilic grain disease #9455 → Add MONDO:7770002 argyrophilic grain disease #9456 → Add new term MONDO:7770002 for argyrophilic grain disease #9482

All were closed. The reviewer repeatedly asked to "update this PR instead of creating a new one" but the agent kept creating new PRs.

2. Annotation Format Misunderstanding

The agent consistently failed to understand MONDO's annotation patterns:

• Grouped multiple PMIDs in one annotation instead of separate entries
• Confused PMC IDs with PMIDs
• Used wrong source annotations for SubClassOf relationships

3. Not Checking for Existing Work

Multiple PRs were closed because the work was already done or in progress by human contributors.

[dragon-ai-agent]

Training Plan: dragon-ai-agent for MONDO Repository

Executive Summary

Analysis of 26 PRs from dragon-ai-agent revealed a 44% success rate (11 merged, 14 closed, 1 open). The failures cluster around a few key behavioral patterns that can be addressed through targeted training.

Caveat: At least one failure (PR #9167) was likely due to later human edits (Class/Individual conflict). Treat this as a validation/process gap, not solely an agent error.

Failure Mode Analysis

Rank	Failure Mode	Occurrences	Impact
1	Creating new PRs instead of updating existing	9	Critical
2	Wrong annotation/reference format	7	High
3	Duplicate of existing work	3	Medium
4	Destructive edits (removal instead of addition)	2	High
5	Failed to follow reviewer instructions	5	High
6	Technical OWL/OBO structure errors	1	Medium

---

Training Instructions

1. CRITICAL: Always Update Existing PRs

Problem: The agent repeatedly created new PRs when asked to fix issues in existing ones. The argyrophilic grain disease saga resulted in 7 PRs for 1 term.

Training Rule:

WHEN a reviewer requests changes to a PR:
  1. ALWAYS commit changes to the SAME PR branch
  2. NEVER create a new PR unless explicitly told to "close this and start fresh"
  3. If you see "please update this PR" - that means SAME PR, not new one
  4. Push to the existing branch, not a new branch

Verification Step:
Before creating any new PR, ask yourself:

• Is there an existing PR for this issue?
• Has a reviewer asked for changes to an existing PR?
• If yes to either, UPDATE the existing PR.

---

2. Learn MONDO Annotation Patterns

Problem: The agent consistently grouped PMIDs together instead of creating individual annotations.

Training Rule - Individual Annotations:

# WRONG - grouped PMIDs:
synonym: "AGD" EXACT [PMID:29213935, PMID:16319301, PMID:18234698]
is_a: MONDO:0001627 {source="PMID:29213935, PMID:16319301"}

# CORRECT - individual annotations:
synonym: "AGD" EXACT [PMID:29213935]
xref: PMID:16319301
xref: PMID:18234698
is_a: MONDO:0001627 {source="PMID:29213935"}
is_a: MONDO:0001627 {source="PMID:16319301"}

Training Rule - Reference Conversion:

ALWAYS convert references to PMID format:
- PMC5618985 → Look up correct PMID (PMID:29213935)
- DOI:10.1093/brain/awm305 → Look up PMID (PMID:18234698)
- DOI:10.1097/00019442-200512000-00008 → Look up PMID (PMID:16319301)

Use PubMed or the DOI resolver to get accurate PMIDs.
NEVER guess PMID numbers.

Training Rule - Source Annotations:

For SubClassOf/is_a relationships:
- Use authoritative sources: OMIM, PMID, ORPHA
- NOT GitHub issue URLs
- Example: {source="OMIM:620836"} not {source="https://github.com/..."}

---

3. Check for Existing Work Before Starting

Problem: Multiple PRs were closed because work was already done or in progress.

Training Rule - Pre-Work Checklist:

BEFORE starting any PR:
1. Search for existing PRs:
   gh pr list --search "<keywords>" --repo monarch-initiative/mondo --state all

2. Check issue comments for:
   - Recent activity
   - "Fixed by #XXXX" references
   - Other contributors already assigned

3. Check issue labels:
   - "in progress" or similar
   - "assigned" to someone

4. If in doubt, comment on the issue:
   "Is anyone currently working on this? I'd like to help."

---

4. Additive, Not Destructive Edits

Problem: The agent removed existing relationships instead of adding new ones alongside them.

Training Rule:

In MONDO, the default behavior is ADDITIVE:
- ADD new is_a relationships; don't REMOVE existing ones
- ADD new annotations; don't REPLACE unless told to
- When simplifying, only remove what is EXPLICITLY requested

Exception: Only remove content if the issue/reviewer specifically says:
- "remove this relationship"
- "delete this annotation"
- "obsolete this term"

Verification Step:
Before committing, review the diff:

• Are you removing any is_a relationships? If yes, confirm this was requested.
• Are you removing any annotations? If yes, confirm this was requested.

---

5. Follow Reviewer Instructions Precisely

Problem: The agent made partial fixes or misinterpreted reviewer requests.

Training Rule:

When a reviewer provides feedback:
1. Copy each instruction into a checklist
2. Address EACH point explicitly
3. If an instruction is unclear, ask for clarification BEFORE implementing
4. In your response, show exactly how each point was addressed

Example response format:
"I have made the following changes:
1. ✅ [Instruction 1] - Changed X to Y
2. ✅ [Instruction 2] - Added annotation Z
3. ❓ [Instruction 3] - Unclear, could you clarify?"

Training Rule - Ask for Examples:

If told to "look at other terms for examples":
1. Actually search for similar terms in the ontology
2. Copy the EXACT pattern used
3. Show the example in your commit message or PR comment

---

6. Learn MONDO Obsoletion Patterns

Problem: Wrong obsoletion reasons and codes were used.

Training Rule - Obsoletion Codes:

Standard IAO codes for term obsoletion:

For merged terms (duplicates):
  property_value: IAO:0000231 MONDO:TermsMerged
  replaced_by: MONDO:XXXXXXX

For out-of-scope terms:
  property_value: IAO:0000231 MONDO:OutOfScope

For duplicate in MONDO:
  property_value: IAO:0000231 MONDO:duplicateInMondo

Always include issue tracker reference:
  property_value: IAO:0000233 "https://github.com/monarch-initiative/mondo/issues/XXXX"

---

7. Validate Before Committing

Problem: OWL structure errors and syntax issues.

Training Rule:

Before pushing any commit:
1. Run: robot validate src/ontology/mondo-edit.obo
2. Run: robot convert -i src/ontology/mondo-edit.obo -o /dev/null
3. Check for Class vs Individual issues

If validation fails, FIX before pushing.

---

Behavioral Guidelines

Communication Pattern

When receiving a task:
1. Acknowledge the task
2. State your plan
3. Ask clarifying questions if needed
4. Implement
5. Show exactly what you did
6. Ask if anything needs adjustment

DO NOT:
- Claim you've made changes you haven't made
- Mark a review as "approved" (you're the contributor, not reviewer)
- Create duplicate PRs
- Ignore feedback

Handling Feedback

When receiving review feedback:
1. Thank the reviewer
2. Address each point systematically
3. If you disagree, explain why politely
4. Commit to the SAME PR
5. Request re-review when done

Example:
"Thank you for the feedback! I've addressed each point:
1. [Point 1] - Done, see line 123
2. [Point 2] - Done, see line 456
3. [Point 3] - I have a question: [clarification]

Ready for re-review when you have time."

When to Stop and Ask

STOP and ask a human when:
- You've created more than 2 PRs for the same issue
- A reviewer seems frustrated
- You're not sure if you understood the instructions
- You're about to delete significant content
- The issue is assigned to someone else

---

Success Patterns to Reinforce

The following PR types had high success rates:

1. Simple parent additions (Add infectious disease as parent to renal infectious disease #9717, Add logical definition to renal infectious disease (MONDO:0004369) #9724, Add viral respiratory tract infection as parent to common cold #9726)

   • Adding one is_a relationship
   • Clear issue request
   • Minimal changes

2. Cross-reference additions (Add HP:0003002 (Breast carcinoma) xref to MONDO:0004989 (breast carcinoma) #9283, Add cutaneous solitary mastocytoma as subclass of cutaneous mastocytoma #9198, Add MONDO:0005341 (skin basal cell carcinoma) as parent to MONDO:0019317 #9652)

   • Adding xrefs or synonyms
   • Well-defined patterns

3. URL/link fixes (Fix CDC URLs - addressing broken/redirected links #9261)

   • Mechanical changes with clear before/after

4. Follow-up fixes (Add additional SubClassOf relationship for MONDO:0970994 to MONDO:0800145 #9209)

   • Learning from previous failed attempt
   • Applying feedback correctly

---

Metrics for Improvement

Target metrics after training:

Metric	Current	Target
Overall success rate	44%	75%
PRs per issue	2.3	1.2
"Update existing PR" compliance	0%	100%
Correct annotation format	~30%	95%
First-attempt success	~40%	60%

---

Implementation Checklist

For the agent developer/trainer:

• Add pre-PR check: search for existing PRs
• Add pre-commit validation: robot validate
• Add instruction parsing: create checklist from reviewer feedback
• Add branch awareness: detect when to update vs create
• Add pattern library: MONDO annotation examples
• Add reference converter: PMC/DOI to PMID lookup
• Add failure detection: recognize repeated mistakes

---

Appendix: Quick Reference Card

MONDO Agent Quick Reference
===========================

BEFORE creating a PR:
□ Search for existing PRs on this issue
□ Check if anyone is already working on it
□ Read the full issue including all comments

WHEN making changes:
□ ADD relationships, don't REMOVE (unless explicit)
□ Use PMID format, not PMC or DOI
□ Individual annotations, not grouped
□ Use OMIM/PMID for sources, not GitHub URLs

WHEN receiving feedback:
□ Create checklist from reviewer comments
□ Address EACH point
□ Update SAME PR, don't create new one
□ Ask for clarification if unsure

BEFORE pushing:
□ Run robot validate
□ Check you're on the correct branch
□ Review diff for unintended deletions

IF struggling:
□ Stop after 2 failed PRs on same issue
□ Ask for a concrete example
□ Request human review of approach