Skip to content

Commit 059e0a8

Browse files
perf regression guard detox (#9835)
* perf-ci: add Detox-based perf regression guard - Add iOS/Android Detox configs and e2e perf regression test (3-run loop) - Add perf-ci runners (debug/release/daemon), thresholds, Slack notification - Improve performance-server: /api/health, session overview index - Add docs and launchd templates * Fix lint and unit test issue. * fix(android): pin detox gradle dependency * docs(perf-ci): note to pin Android detox dependency --------- Co-authored-by: huhuanming <[email protected]>
1 parent 700fc8a commit 059e0a8

38 files changed

+4603
-38
lines changed
Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
---
2+
name: perf-optimizer
3+
description: "Systematic performance optimization and regression debugging for OneKey mobile app (iOS). Use when: (1) Fixing performance regressions - when metrics like tokensStartMs, tokensSpanMs, or functionCallCount have regressed and need to be brought back to normal levels, (2) Improving baseline performance - when there's a need to optimize cold start time or reduce function call overhead, (3) User requests performance optimization/improvement/debugging for the app's startup or home screen refresh flow."
4+
---
5+
6+
# Performance Optimizer
7+
8+
Systematic workflow for diagnosing and fixing performance issues in the OneKey mobile app using the perf-ci infrastructure and performance-server tooling.
9+
10+
## Overview
11+
12+
This skill provides a structured iterative approach to:
13+
- Establish performance baselines from existing sessions
14+
- Run controlled perf measurements (3 runs, median aggregation)
15+
- Analyze session data to identify bottlenecks
16+
- Make targeted code changes
17+
- Verify improvements against thresholds
18+
- Document all changes and results
19+
20+
**Key Metrics:**
21+
- `tokensStartMs`: Time when Home tokens refresh starts (lower is better)
22+
- `tokensSpanMs`: Duration of Home tokens refresh (lower is better)
23+
- `functionCallCount`: Total function calls during session (lower is better)
24+
25+
**Success Criteria:**
26+
-**SUCCESS**: Time metrics improve by ≥10%
27+
- 🌟 **MINOR_IMPROVEMENT**: Time unchanged but function calls reduce by ≥20% (safe, small-scope changes)
28+
-**NO_IMPROVEMENT**: Neither threshold met → revert changes
29+
30+
## Workflow
31+
32+
### Phase 1: Setup and Baseline
33+
34+
#### Step 1.1: Select Baseline Session
35+
36+
Ask user to choose a baseline session or help them select one:
37+
38+
```bash
39+
# List recent sessions with key metrics
40+
cat ~/perf-sessions/sessions.overview.jsonl | \
41+
jq -r '[.sessionId, .createdAt, .marks["Home:refresh:done:tokens"]] | @tsv' | \
42+
tail -20
43+
```
44+
45+
User can specify:
46+
- A known good session (for regression fixes)
47+
- A recent session (for improvement work)
48+
- Let you choose a representative session
49+
50+
#### Step 1.2: Analyze Baseline
51+
52+
Extract baseline metrics from the session:
53+
54+
```bash
55+
# Get detailed analysis
56+
node development/performance-server/cli/derive-session.js <baseline-sessionId> \
57+
--pretty \
58+
--output /tmp/perf-baseline-derived.json
59+
```
60+
61+
Read baseline metrics from `~/perf-sessions/<sessionId>/mark.log`:
62+
63+
```bash
64+
# Extract tokensStartMs (timestamp of Home:refresh:start:tokens)
65+
grep "Home:refresh:start:tokens" ~/perf-sessions/<sessionId>/mark.log | jq '.timestamp'
66+
67+
# Extract tokensSpanMs (done - start)
68+
grep "Home:refresh:done:tokens" ~/perf-sessions/<sessionId>/mark.log | jq '.timestamp'
69+
70+
# Count function calls
71+
wc -l < ~/perf-sessions/<sessionId>/function_call.log
72+
```
73+
74+
Create baseline metrics JSON for comparison:
75+
76+
```bash
77+
echo '{"tokensStartMs": <start>, "tokensSpanMs": <span>, "functionCallCount": <count>}' > /tmp/baseline-metrics.json
78+
```
79+
80+
#### Step 1.3: Initialize Documentation
81+
82+
Create session document at `development/output/perf-optimization-<timestamp>.md` using the template from `references/template.md`. Fill in:
83+
- Current date/time
84+
- Baseline session ID
85+
- Baseline metrics
86+
- Current branch name
87+
- Target (regression fix or improvement)
88+
89+
### Phase 2: Iterative Optimization Loop
90+
91+
**Maximum iterations: 10**
92+
93+
For each iteration (run in a sub-agent):
94+
95+
#### Step 2.1: Run Performance Tests
96+
97+
The perf script **automatically runs 3 times** and aggregates results:
98+
99+
```bash
100+
node development/perf-ci/run-ios-perf-detox-release.js
101+
```
102+
103+
Output location: `development/perf-ci/output/<jobId>/`
104+
- `report.json` - Contains aggregated results in `agg` field
105+
- `detox/runs.json` - Contains individual run sessionIds
106+
107+
Extract current metrics from `report.json`:
108+
109+
```bash
110+
# Read aggregated metrics directly
111+
cat development/perf-ci/output/<jobId>/report.json | jq '{
112+
tokensStartMs: .agg.tokensStartMs,
113+
tokensSpanMs: .agg.tokensSpanMs,
114+
functionCallCount: .agg.functionCallCount
115+
}' > /tmp/current-metrics.json
116+
```
117+
118+
#### Step 2.2: Analyze Current Performance
119+
120+
For deeper analysis, run derive-session on individual sessions:
121+
122+
```bash
123+
# Get sessionIds from the run
124+
SESSIONS=$(cat development/perf-ci/output/<jobId>/detox/runs.json | jq -r '.runs[].sessionId')
125+
126+
# Analyze each session
127+
for sid in $SESSIONS; do
128+
node development/performance-server/cli/derive-session.js $sid \
129+
--pretty \
130+
--output /tmp/perf-derived-$sid.json
131+
done
132+
```
133+
134+
Focus on these sections in the derived output:
135+
- **slowFunctions**: Functions taking the most cumulative time
136+
- **homeRefreshTokens**: What's consuming time in the critical refresh window
137+
- **jsblock**: Main thread blocks causing delays
138+
- **repeatedCalls**: Thrashing patterns or excessive re-renders
139+
- **keyMarks**: Critical milestone timing
140+
141+
Identify top 1-3 bottlenecks that are:
142+
- Taking significant time
143+
- Potentially optimizable
144+
- Within the critical path (Home refresh flow)
145+
146+
#### Step 2.3: Determine Action
147+
148+
Compare current metrics to baseline:
149+
150+
```bash
151+
# Quick comparison
152+
cat /tmp/baseline-metrics.json
153+
cat /tmp/current-metrics.json
154+
155+
# Calculate deltas manually or use script in skill directory
156+
```
157+
158+
**Decision tree:**
159+
160+
If current metrics show improvement over baseline:
161+
-**SUCCESS** (≥10% time improvement) → **STOP**, document success
162+
- 🌟 **MINOR_IMPROVEMENT** (≥20% function call reduction, time stable) → Create branch, commit, return to main branch, continue
163+
164+
If no improvement yet:
165+
- Continue to Step 2.4 (make changes)
166+
167+
If iteration count reaches 10:
168+
- Document findings and stop
169+
170+
#### Step 2.4: Make Code Changes
171+
172+
Based on analysis, make **ONE** targeted change per iteration:
173+
174+
**Change types:**
175+
1. **Optimization**: Remove redundant work, cache results, reduce allocations
176+
2. **Add perfMark**: Add marks to understand unclear bottlenecks better
177+
3. **Both**: Add marks + optimize in same area
178+
179+
**Guidelines:**
180+
- One change at a time (unless analysis proves multiple changes must work together)
181+
- Small, focused changes
182+
- Safe changes only (never break functionality)
183+
- Document rationale clearly
184+
185+
**Adding perfMarks:**
186+
187+
Use the performance utilities in `packages/shared/src/performance/`:
188+
189+
```typescript
190+
import { perfMark } from '@onekeyhq/shared/src/performance/perfMark';
191+
192+
// Add mark at a specific point
193+
perfMark('MyComponent:operation:start');
194+
// ... operation ...
195+
perfMark('MyComponent:operation:done');
196+
```
197+
198+
Naming convention: `<Component>:<action>:<phase>` (e.g., `Home:refresh:start:tokens`)
199+
200+
**If adding perfMarks for investigation:**
201+
1. Add marks around suspected bottleneck
202+
2. Run one perf cycle with marks
203+
3. Analyze new data with marks visible
204+
4. Then make code optimization
205+
5. Verify with another perf cycle
206+
207+
#### Step 2.5: Document Iteration
208+
209+
Update the session document with:
210+
- **Analysis**: Job ID, session IDs, median metrics, key findings from derive-session
211+
- **Code Changes**: File, location, change type, description, rationale
212+
- **Verification Results**: New job ID, metrics, deltas vs previous/baseline, verdict, action taken
213+
214+
### Phase 3: Finalization
215+
216+
#### Step 3.1: Handle MINOR_IMPROVEMENT Branch
217+
218+
If any iterations resulted in MINOR_IMPROVEMENT:
219+
220+
```bash
221+
git checkout -b perf/minor-<description>
222+
git add <changed-files>
223+
git commit -m "perf: <description>
224+
225+
Reduces function call count by X% while maintaining time metrics.
226+
227+
Reason: <brief explanation>
228+
"
229+
git checkout <original-branch>
230+
git restore .
231+
```
232+
233+
Document the branch name in the session document.
234+
235+
#### Step 3.2: Complete Documentation
236+
237+
Fill in the Summary section:
238+
- Total iterations run
239+
- Final result (SUCCESS with % improvement, or still investigating)
240+
- List all effective changes
241+
- List all ineffective changes with reasons
242+
- List any branches created
243+
- Next steps if incomplete
244+
245+
## Key Files and Paths
246+
247+
**Perf Infrastructure:**
248+
- `development/perf-ci/run-ios-perf-detox-release.js` - Main perf runner
249+
- `development/perf-ci/output/<jobId>/` - Job output directory
250+
- `development/performance-server/cli/derive-session.js` - Session analyzer
251+
- `~/perf-sessions/` - Session data storage (default)
252+
- `~/perf-sessions/sessions.overview.jsonl` - Session index
253+
254+
**Thresholds:**
255+
- `development/perf-ci/thresholds/ios.release.json` - Release mode thresholds
256+
257+
**Performance Utilities:**
258+
- `packages/shared/src/performance/perfMark.ts` - Performance marking utility
259+
260+
## References
261+
262+
- **references/template.md**: Session documentation template
263+
- **references/perf_tool_guide.md**: Detailed guide to derive-session and analysis tools
264+
265+
## Important Notes
266+
267+
1. **Run each optimization loop in a sub-agent** to avoid context bloat
268+
2. **Never commit changes unless** SUCCESS or MINOR_IMPROVEMENT
269+
3. **Always document failed attempts** - helps avoid repeating ineffective changes
270+
4. **Trust the data** - if metrics don't improve, revert even if change "should" help
271+
5. **Be patient** - each perf run takes significant time (build + 3 runs); rushing leads to mistakes
272+
6. **Focus on the critical path** - Home screen tokens refresh is the key metric
273+
7. **Watch for trade-offs** - some optimizations might reduce one metric but increase another
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Performance Monitoring Tool Guide
2+
3+
## Overview
4+
5+
The performance monitoring tooling consists of:
6+
- `development/performance-server/`: WebSocket server + Dashboard + Analysis libraries
7+
- `packages/shared/src/performance/collectors/`: Performance data collectors
8+
- Session data storage: `~/perf-sessions/` (default)
9+
10+
## Key Command: derive-session
11+
12+
Generate analysis report from a session ID:
13+
14+
```bash
15+
node development/performance-server/cli/derive-session.js <sessionId> \
16+
--pretty \
17+
--output /tmp/perf-<sessionId>.json
18+
```
19+
20+
### Common Options
21+
22+
- `--topSlow 50`: Number of slowFunctions to output (default: 50)
23+
- `--topRepeated 50`: Number of repeatedCalls to output (default: 50)
24+
- `--fpsThreshold 10`: Low FPS threshold (<10 = stuttering window), use `--noFps` to disable
25+
- `--fpsTopWindows 10`: Output top N worst FPS windows (default: 10)
26+
- `--fpsTopFunctions 25`: Top N hot functions per FPS window (default: 25)
27+
- `--noJsblock`: Disable jsblock window analysis
28+
29+
## Output Structure
30+
31+
The derived JSON contains:
32+
33+
### lowFps
34+
FPS windows below threshold with:
35+
- `topWindows`: Worst performing time windows
36+
- Hot functions in each window (by total time/call count)
37+
38+
### jsblock
39+
Main thread blocking events:
40+
- `topWindows`: jsblock:main windows (JS thread stalls)
41+
- Hot functions causing the blocks
42+
43+
### keyMarks
44+
Critical milestone marks:
45+
- `Home:refresh:start:tokens` / `Home:refresh:done:tokens`
46+
- `Home:done:tokens`
47+
- `AllNet:*` phase marks (for explaining delays/jsblock)
48+
- `Bootstrap:*` / `Home:defi:*` marks (for staggering verification)
49+
50+
### homeRefreshTokens
51+
Analysis of the `Home:refresh:start:tokens → Home:refresh:done:tokens` window:
52+
- Top functions + bgcall/storage/simpledb/jsblock marks
53+
- Directly answers "what's consuming time in refresh span"
54+
55+
### slowFunctions
56+
Session-wide aggregated slow function list:
57+
- Functions sorted by total execution time
58+
- Call counts and average duration
59+
60+
### repeatedCalls
61+
High-frequency repeated calls in short time windows:
62+
- Used to locate "thrashing/re-renders/loop triggers"
63+
- Indicates potential optimization opportunities
64+
65+
### repeatedCallsOverall
66+
Overall call count ranking:
67+
- Most frequently called functions across entire session
68+
- Helps identify hot paths
69+
70+
## Session Data Location
71+
72+
Default: `~/perf-sessions/<sessionId>/`
73+
74+
Each session directory contains:
75+
- `mark.log`: Performance marks (JSONL format)
76+
- `function_call.log`: Function call traces (JSONL format)
77+
- Other collector logs
78+
79+
## Reading sessions.overview.jsonl
80+
81+
To list all available sessions:
82+
83+
```bash
84+
cat ~/perf-sessions/sessions.overview.jsonl | jq -r '[.sessionId, .createdAt, .marks["Home:refresh:done:tokens"]] | @tsv'
85+
```
86+
87+
Each line is a JSON object with session metadata and key marks.

0 commit comments

Comments
 (0)