Commit d876391
authored
ci: adapt automated tests metrics logic (#27552)
<!--
Please submit this PR as a draft initially.
Do not mark it as "Ready for review" until the template has been
completely filled out, and PR status checks have passed at least once.
-->
## **Description**
Updates QA stats collection to emit richer, more explicit metrics.
Example:
```
✅ QA stats written to ./qa-stats.json: {
unit: {
total_tests_run: 42997,
predict_tests_run: 3604,
selectors_tests_run: 1141,
core_tests_run: 5153,
other_tests_run: 4923,
store_tests_run: 1047,
tokens_tests_run: 240,
bridge_tests_run: 1796,
earn_tests_run: 1519,
rewards_tests_run: 2037,
util_tests_run: 2328,
perps_tests_run: 5439,
networksmanagement_tests_run: 234,
reducers_tests_run: 641,
card_tests_run: 2738,
homepage_tests_run: 284,
controllers_tests_run: 1440,
confirmations_tests_run: 2502,
components_base_tests_run: 323,
components_hooks_tests_run: 784,
settings_tests_run: 229,
browsertab_tests_run: 232,
multichainaccounts_tests_run: 356,
ramp_tests_run: 2128,
stake_tests_run: 207,
component_library_tests_run: 1006,
trending_tests_run: 325,
total_tests_defined: 40267,
total_tests_skipped: 17,
line_coverage: 87.4,
statement_coverage: 87,
branch_coverage: 79.1,
function_coverage: 83.3
},
component_view: {
total_tests_run: 104,
perps_tests_run: 32,
earn_tests_run: 28,
confirmations_tests_run: 5,
predict_tests_run: 17,
wallet_tests_run: 5,
assetdetails_tests_run: 1,
bridge_tests_run: 9,
trendingview_tests_run: 6,
walletactions_tests_run: 1,
total_tests_defined: 102,
total_tests_skipped: 0,
line_coverage: 15.8,
statement_coverage: 15.7,
branch_coverage: 9.4,
function_coverage: 13
},
e2e: {
main_tests_run: 140,
main_android_tests_run: 140,
main_ios_tests_run: 140,
flask_tests_run: 72,
flask_android_tests_run: 72,
flask_ios_tests_run: 72,
total_tests_run: 212,
multichain_api_tests_run: 12,
prediction_market_tests_run: 10,
wallet_platform_tests_run: 35,
network_expansion_tests_run: 10,
confirmations_tests_run: 31,
trade_tests_run: 12,
identity_tests_run: 6,
network_abstraction_tests_run: 12,
accounts_tests_run: 2,
ramps_tests_run: 3,
card_tests_run: 6,
perps_tests_run: 1,
total_tests_defined: 225,
total_tests_skipped: 18
},
performance: {
total_tests_defined: 22,
total_tests_skipped: 1,
login_tests_defined: 11,
mm_connect_tests_defined: 6,
onboarding_tests_defined: 5
}
}
```
<!--
Write a short description of the changes included in this pull request,
also include relevant motivation and context. Have in mind the following
questions:
1. What is the reason for the change?
2. What is the improvement/solution?
-->
## **Changelog**
<!--
If this PR is not End-User-Facing and should not show up in the
CHANGELOG, you can choose to either:
1. Write `CHANGELOG entry: null`
2. Label with `no-changelog`
If this PR is End-User-Facing, please write a short User-Facing
description in the past tense like:
`CHANGELOG entry: Added a new tab for users to see their NFTs`
`CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker`
(This helps the Release Engineer do their job more quickly and
accurately)
-->
CHANGELOG entry:
## **Related issues**
Fixes:
## **Manual testing steps**
```gherkin
Feature: my feature name
Scenario: user [verb for user action]
Given [describe expected initial app state]
When user [verb for user action]
Then [describe expected outcome]
```
## **Screenshots/Recordings**
<!-- If applicable, add screenshots and/or recordings to visualize the
before and after of your change. -->
### **Before**
<!-- [screenshots/recordings] -->
### **After**
<!-- [screenshots/recordings] -->
## **Pre-merge author checklist**
- [x] I've followed [MetaMask Contributor
Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile
Coding
Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've completed the PR template to the best of my ability
- [x] I've included tests if applicable
- [x] I've documented my code using [JSDoc](https://jsdoc.app/) format
if applicable
- [x] I've applied the right labels on the PR (see [labeling
guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)).
Not required for external contributors.
## **Pre-merge reviewer checklist**
- [x] I've manually tested the PR (e.g. pull and build branch, run the
app, test code being changed).
- [x] I confirm that this PR addresses all acceptance criteria described
in the ticket it closes and includes the necessary testing evidence such
as recordings and or screenshots.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Moderate risk because it changes how QA metrics are keyed/collected
and updates CI artifact production/retention; mismatches in artifact
names or parsing could silently drop metrics or skew dashboards.
>
> **Overview**
> Updates the QA stats pipeline to generate **richer, explicitly named
metrics** (e.g., `*_tests_run`, `total_tests_defined`,
`total_tests_skipped`) and adds static source scanning to estimate
defined/skipped tests across unit, component-view, E2E smoke, and
performance suites.
>
> The collector no longer depends on a passed `WORKFLOW_RUN_ID`; it now
resolves the **latest successful `ci.yml` run on `main`** via the GitHub
API and pulls additional artifacts to include **coverage percentages**
for unit and component-view tests.
>
> CI workflows are adjusted to support this by producing/uploading new
coverage-summary artifacts and setting **7-day retention** on several
uploaded artifacts, and the `qa-stats.yml` workflow switches from
`workflow_run` triggering to a **daily scheduled run**.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
ea63d48. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->1 parent 1db9700 commit d876391
7 files changed
Lines changed: 292 additions & 45 deletions
0 commit comments