-
Notifications
You must be signed in to change notification settings - Fork 285
Chore: Review script to review long pull requests #6318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Lines 9-18: Added Dafny Scripts folder detection Lines 20-25: Auto-detect PR number from current branch pattern Line 85: Added MAX_RETRIES=5 configuration Lines 91-110: Modified repository handling for Dafny detection Lines 111-114: Added PR title fetching and display Lines 340-342: Added PR title to the AI prompt Lines 344-375: Implemented retry logic with proper error handling Lines 376-380: Enhanced error messages with retry count Lines 540-570: Conditional cleanup - only delete temp folder on success, preserve on failure
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you use bash instead of C# to define this script? I think it would be easier to maintain if it was C#, because the repository already requires reading C# and debugging C# is easier than bash.
I asked Q to develop me a script to do that, so this script itself is an iteration of an interaction with Q. I haven't asked specifically for bash. |
|
FWIW, I found it helpful for reviewing #3324 (and finding things to fix in it!), which would be a huge effort otherwise. |
Currently, reviewing long PRs, like PRs that changes multiple expect files, is discouraging.
This PR automates the reviewing process by delegating some portion of reviewing, i.e. low-level reviewing, to an agent such as Amazon Q or Claude Code
What was changed?
I added a script
./review.shthat requires a PR number and automatically performs a review of every change to see if they conform to the PR description.It clones the repository, lists all the files, and input maximum 500 lines of diff into a prompt that asks if the diff is consistent with what we could have expected from the PR description.
Sample output while the script is running live.
How has this been tested?
Exiting PR
I ran this script on #3324 to ensure the PR description is consistent with the changes. It already detected the following issues:
Self-test: Positive case
I also tested this script on this PR itself. Here was the output:
Self-test: Negative case:
When deleting a line in the Makefile, we immediately get meaningful feedback
and then when the analysis is finished, we see
By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.