Skip to content

tests: add github KPIs script - #17024

Open
maykathm wants to merge 9 commits into
canonical:masterfrom
maykathm:add-github-kpis-script
Open

tests: add github KPIs script#17024
maykathm wants to merge 9 commits into
canonical:masterfrom
maykathm:add-github-kpis-script

Conversation

@maykathm

@maykathm maykathm commented May 5, 2026

Copy link
Copy Markdown
Contributor

The idea behind this script is to generate interesting data points of each PR merged in a given time frame. Once generated, then noteworthy metrics can be generated on the output json file trivially with jq. The script's help contains some example interesting jq queries.

This script collects data for each PR in a given time frame:

  • number of attempts on ci-tests.yaml before merging
  • merged timestamp
  • whether spread tests ran
  • whether nested tests ran
  • whether the PR was force merged
  • number of skipped tests (not counting variants)
  • total runtime across all attempts
  • runtime in first attempt
  • if the first attempt only included fundamental systems
  • total number of spread tests that ran in the PR (not counting retrys)

Example usage

$ ./github-kpis.sh --start '2026-05-02' --all > 2026-05-02.json
Fetching PRs merged between 2026-05-02 and now...                                                                                                                                             
PRs fetched: 8                                                                                                                                                                                
Fetching number of attempts for each PR...                                                                                                                                                    
Attempts progress: [##############################] 8/8                                                                                                                                       
Done.                                                                                                                                                                                         
Determining whether each PR was force merged...                                                                                                                                               
Force-merged progress: [##############################] 8/8                                                                                                                                   
Done.                                                                                                                                                                                         
Determining number of skipped tests for each PR...                                                                                                                                            
Skipped tests progress: [##############################] 8/8                                                                                                                                  
Done.                                                                                                                                                                                         
Calculating total runtime for each PR...                                                                                                                                                      
Runtime progress: [##############################] 8/8                                                                                                                                        
Done.                                                                                                                                                                                         
Calculating test totals for each PR...                                                                                                                                                        
Test totals progress: [##############################] 8/8                                                                                                                                    
Done.

https://warthogs.atlassian.net/browse/SNAPDENG-36830

@maykathm maykathm added the Skip spread Indicate that spread job should not run label May 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a new Bash utility under tests/utils/ to collect CI/test KPIs for PRs merged within a given time range (or resumed from previously collected JSON) by querying GitHub via gh and processing results with jq.

Changes:

  • Added tests/utils/github-kpis.sh to fetch merged PRs in a date range and augment them with KPI fields (attempt count, forced-merge detection, skipped tests, runtime totals, and test totals).
  • Implemented progress reporting and staged execution (run_stage) to allow partial results and resume flows.

Comment thread tests/utils/github-kpis.sh Outdated
Comment thread tests/utils/github-kpis.sh Outdated
Comment thread tests/utils/github-kpis.sh Outdated
Comment thread tests/utils/github-kpis.sh Outdated
Comment thread tests/utils/github-kpis.sh Outdated
Comment thread tests/utils/github-kpis.sh Outdated
@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown

Thu Jun 4 19:13:05 UTC 2026

Spread tests skipped

maykathm added 3 commits May 6, 2026 09:16
add trap for tmpdir cleanup
fix unset variable
change exit to return
add comment explaining fundamental systems check
fix PR loop when there are more than 1000 PRs
@maykathm
maykathm force-pushed the add-github-kpis-script branch 2 times, most recently from 946a281 to fe062ce Compare May 7, 2026 09:46
@codecov

codecov Bot commented May 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.15%. Comparing base (6ef8a8e) to head (f160c7d).
⚠️ Report is 170 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #17024      +/-   ##
==========================================
+ Coverage   79.04%   79.15%   +0.11%     
==========================================
  Files        1375     1374       -1     
  Lines      191012   192728    +1716     
  Branches     2465     2466       +1     
==========================================
+ Hits       150981   152562    +1581     
- Misses      30924    30993      +69     
- Partials     9107     9173      +66     
Flag Coverage Δ
unittests 79.15% <ø> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@maykathm
maykathm requested a review from ndyer May 22, 2026 08:43
@maykathm
maykathm force-pushed the add-github-kpis-script branch from 0fc8962 to 509a454 Compare May 22, 2026 16:07
@maykathm
maykathm force-pushed the add-github-kpis-script branch from 509a454 to b259b6a Compare May 22, 2026 16:23
return "external"

permission = gh_request("api", f"/repos/canonical/snapd/collaborators/{login}/permission", "--jq", ".permission", allow_failure=True)
if permission and permission.strip() in {"admin", "maintain", "write"}:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we don't to include snapd guests here? In which case we should remove "write"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense for us to consider snapd guests as external?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it probably does. Just to double check @ndyer what do you prefer? Do you want snapd guests to be considered "snapd" or "external"?

@sergiocazzolato

Copy link
Copy Markdown
Contributor

I have added some changes in order to fix the filter by dates and to reduce the amount of calls to the gh api.
I see the total number of tests had 0 when I ran the queries, I still not sure if we are measuring properly that, and if we could avoid downloading all the artifacts which makes the process very slow.

@maykathm

maykathm commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

I have added some changes in order to fix the filter by dates and to reduce the amount of calls to the gh api. I see the total number of tests had 0 when I ran the queries, I still not sure if we are measuring properly that, and if we could avoid downloading all the artifacts which makes the process very slow.

Was the total number of 0 you saw for PRs that skipped spread? Those should all be 0. Just in local testing, the number of tests I see appears correct (though I need to manually check to make sure the numbers are effectively correct).

One alternative to downloading the spread-results json files is to parse the logs from each job. I originally discarded that approach since it was extremely slow in my local testing and resulted in many 500s. It requires going job by job since the logs are too large for a single API request.

We could also remove the total number of spread tests run if it's too problematic. Though if we run the script every day, it shouldn't be too bad.

@maykathm
maykathm requested a review from Rnfudge02 June 18, 2026 11:21
@Rnfudge02

Rnfudge02 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

I ran the example command using --all.

I tried the example of your script, but 4 fields show up empty for all 181 PRs the script grabs, they are:

  • databaseId
  • total-runtime-minutes
  • first-attempt-minutes
  • fire-attempt-only-fundamental

I don't think all of these would be tagged to skip spread, so I think the error must be in the ensure_metadata() function, or the databaseId cannot be retrieved

@Rnfudge02

Copy link
Copy Markdown
Contributor

The rest of the metrics work well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Skip spread Indicate that spread job should not run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants