tests: add github KPIs script - #17024
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new Bash utility under tests/utils/ to collect CI/test KPIs for PRs merged within a given time range (or resumed from previously collected JSON) by querying GitHub via gh and processing results with jq.
Changes:
- Added
tests/utils/github-kpis.shto fetch merged PRs in a date range and augment them with KPI fields (attempt count, forced-merge detection, skipped tests, runtime totals, and test totals). - Implemented progress reporting and staged execution (
run_stage) to allow partial results and resume flows.
|
Thu Jun 4 19:13:05 UTC 2026 Spread tests skipped |
add trap for tmpdir cleanup fix unset variable change exit to return add comment explaining fundamental systems check fix PR loop when there are more than 1000 PRs
946a281 to
fe062ce
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #17024 +/- ##
==========================================
+ Coverage 79.04% 79.15% +0.11%
==========================================
Files 1375 1374 -1
Lines 191012 192728 +1716
Branches 2465 2466 +1
==========================================
+ Hits 150981 152562 +1581
- Misses 30924 30993 +69
- Partials 9107 9173 +66
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
0fc8962 to
509a454
Compare
509a454 to
b259b6a
Compare
| return "external" | ||
|
|
||
| permission = gh_request("api", f"/repos/canonical/snapd/collaborators/{login}/permission", "--jq", ".permission", allow_failure=True) | ||
| if permission and permission.strip() in {"admin", "maintain", "write"}: |
There was a problem hiding this comment.
Perhaps we don't to include snapd guests here? In which case we should remove "write"
There was a problem hiding this comment.
Would it make sense for us to consider snapd guests as external?
There was a problem hiding this comment.
I think it probably does. Just to double check @ndyer what do you prefer? Do you want snapd guests to be considered "snapd" or "external"?
|
I have added some changes in order to fix the filter by dates and to reduce the amount of calls to the gh api. |
Was the total number of 0 you saw for PRs that skipped spread? Those should all be 0. Just in local testing, the number of tests I see appears correct (though I need to manually check to make sure the numbers are effectively correct). One alternative to downloading the spread-results json files is to parse the logs from each job. I originally discarded that approach since it was extremely slow in my local testing and resulted in many 500s. It requires going job by job since the logs are too large for a single API request. We could also remove the total number of spread tests run if it's too problematic. Though if we run the script every day, it shouldn't be too bad. |
|
I ran the example command using --all. I tried the example of your script, but 4 fields show up empty for all 181 PRs the script grabs, they are:
I don't think all of these would be tagged to skip spread, so I think the error must be in the ensure_metadata() function, or the databaseId cannot be retrieved |
|
The rest of the metrics work well |
The idea behind this script is to generate interesting data points of each PR merged in a given time frame. Once generated, then noteworthy metrics can be generated on the output json file trivially with
jq. The script's help contains some example interestingjqqueries.This script collects data for each PR in a given time frame:
Example usage
https://warthogs.atlassian.net/browse/SNAPDENG-36830