- Detect when the profile goes stale.
- Track cold-start performance over time.
- Catch startup regressions before release.
Proposed approach (Mike's brainstorm)
- Use Firebase Test Lab's free tier. It has a sufficient number of free runs per day to cover scheduled and on-demand use. If it ever goes away or the limits change, we can revisit.
- Pick two reference devicesone "old phone" and one "new phone" representative of typical user hardware.
- Trigger conditions
- Persistence of results
- A wiki page with entries and a script-generated graph of performance over versionCode.
- Posting a comment on the commit line with the latest numbers and a link to git blame showing the last commit that touched baseline-prof.txt plus its commit message (which by convention includes the previous run's numbers).
- For the regenerated profile, the CI job should commit the result and open a PR for human review
For ref:
I know these need to run on device but I work with Firebase a lot, and...they have a free plan which allows for a sufficient (IMHO) number of runs a day for this to run as a scheduled thing for graphs-over-time and/or the occasional manual_dispatch...first column is the "free" plan:
Originally posted by @mikehardy in #20702 (comment)
Proposed approach (Mike's brainstorm)
For ref:
I know these need to run on device but I work with Firebase a lot, and...they have a free plan which allows for a sufficient (IMHO) number of runs a day for this to run as a scheduled thing for graphs-over-time and/or the occasional manual_dispatch...first column is the "free" plan:
Originally posted by @mikehardy in #20702 (comment)