Skip to content

Conversation

walnutwaldo
Copy link

@walnutwaldo walnutwaldo commented Sep 2, 2025

Fix: Use head:devin/ pattern for Devin PR counting (issue aavetis#52)

Summary

Changed Devin PR counting from author:devin-ai-integration[bot] to head:devin/ pattern to match other AI agents and capture PRs created by users on Devin branches. This addresses issue aavetis#52 by ensuring consistent counting methodology across all agents.

Files modified:

  • collect_data.py - Updated search queries for data collection
  • generate_chart.py - Updated query URLs in agent definitions
  • scripts/add_nondraft_final.py - Updated backfill script for consistency
  • README.md - Updated documentation with new search query links

Review & Testing Checklist for Human

3 critical items to verify:

  • Test the new GitHub search queries manually - Visit the updated search URLs in README.md and verify they return the expected Devin PRs (should include both bot-authored PRs and user PRs on devin/ branches)
  • Confirm data.csv changes should be included - The diff shows updated metrics data which may be test artifacts rather than intended changes. Verify if these should be committed or reverted.
  • Validate metrics reasonability - Check if the new PR counts and success rates make sense compared to historical data and align with expectations from issue Fixing Devin PR accounting aavetis/PRarena#52

Notes

  • This represents a breaking change in metrics collection methodology - historical data used author:devin-ai-integration[bot], new data will use head:devin/
  • The script had API timeout issues during testing, so functionality verification via manual query testing is especially important
  • Link to Devin run: https://staging.itsdev.in/sessions/b4b44184bb594ed69c46b72ddd3f4feb
  • Requested by: @walnutwaldo

Copy link

Original prompt from Walden
Open a PR to fix https://github.com/aavetis/PRarena/issues/52 by also checking for head:devin in the counts

Copy link

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@staging-devin-ai-integration staging-devin-ai-integration bot force-pushed the devin/1756763132-fix-head-devin-counts branch from b00101a to e8b3c3b Compare September 2, 2025 07:27
- Drop author:devin-ai-integration[bot] queries as requested in PR feedback
- Keep only head:devin and head:devin/ queries for Devin metrics
- Update combining logic to use max() of just the head queries
- Addresses GitHub comment feedback from walnutwaldo

Co-Authored-By: Walden <[email protected]>
@staging-devin-ai-integration staging-devin-ai-integration bot force-pushed the devin/1756763132-fix-head-devin-counts branch from 32a1956 to 1990b6b Compare September 4, 2025 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant