diff --git a/README.md b/README.md index 92cb11e5..bcdffc80 100644 --- a/README.md +++ b/README.md @@ -312,6 +312,7 @@ Use these directly when steering/debugging: - `/mms-recipe-doctor` — check tool, skill, harness, runtime-context, and fixture/profile readiness before a run. - `/mms-recipe-harness` — install/verify Mobile or Extension runtime harness. - `/mms-recipe-cook` — author/refine the executable recipe. +- `/mms-recipe-performance` — diagnose real-app bottlenecks and prove paired device performance. - `/mms-recipe-quality` — critique recipe/evidence and force weak proof into explicit gaps. - `/mms-recipe-evidence` — format artifacts into reviewer-ready PR text. - `/mms-recipe-wallet-control` — optional wallet/app primitives for setup, navigation, state, and screenshots. diff --git a/domains/agentic/skills/recipe-performance/references/capture-adapter.md b/domains/agentic/skills/recipe-performance/references/capture-adapter.md new file mode 100644 index 00000000..c12f571b --- /dev/null +++ b/domains/agentic/skills/recipe-performance/references/capture-adapter.md @@ -0,0 +1,63 @@ +# Capture adapter contract + +The screen adapter owns checkout, instrumentation, restart and real UI actions. +The kit intentionally supplies no private bridge, navigation recipe or source +patcher. Reuse the flow's existing harness/collector, or prepare one before the +series. Python 3 and an executable adapter are required on the measurement host. + +The driver calls: + +```text +./capture-block.sh STATE REQUESTED_REF ABSOLUTE_BLOCK_DIR ROUNDS +``` + +The adapter must finish the block before returning. On its owned checkout it: + +1. Selects the requested exact commit; restores only its own instrumentation + and original ref on exit/failure. Never reset unrelated files. Apply the same + measurement patch in each state, retain its diff/hash and verify the served + bundle changed as intended. A checkout alone does not reload Metro's bundle. +2. Restarts the same app/dev client, reinstalls probes lost on restart and + verifies the fixture, actual sampler configuration and unobscured control. +3. Settles for the fixed interval and performs one untimed warm-up per scenario. +4. Resets the per-open probe, invokes the real action, then collects every rAF + callback through and beyond the end mark. A wait after tapping is collection + time, not the metric's window. Exit nonzero for setup/capture failures. +5. Writes exactly ROUNDS raw captures per scenario. Keep source/build/device and + sampler readback, invocation and failure logs alongside them. Do not change + product values to manufacture a measurement. + +These are the original public command forms to compose into the adapter, with +the recipe and its `out`/`rounds` parameters defined for the chosen screen: + +```bash +"$HARNESS" call app.lifecycle command=restart --adapter mobile --target "$REPO" --json +"$HARNESS" run "$FLOW_RECIPE" out="$BLOCK_DIR/data" rounds="$ROUNDS" \ + --adapter mobile --target "$REPO" --hud show --artifacts-dir "$BLOCK_DIR/recipe-run" --json +adb -s "$ANDROID_SERIAL" shell input tap "$TAP_X" "$TAP_Y" +adb -s "$ANDROID_SERIAL" shell input keyevent KEYCODE_BACK +``` + +Bind the physical device, Android emulator serial or iOS simulator UDID and the harness target explicitly. For iOS, use the resolved harness/simulator UI actions instead of adb. Do not assume the active device, +coordinates or a recipe parameter exists. The last two commands illustrate UI +actions inside the prepared flow; they are not an extra tap after the recipe. + +Each `data/timed-N.json` is this object, or a JSON string encoding it: + +```json +{ + "marks": [{"n": "press", "t": 1000}, {"n": "open-complete", "t": 2000}], + "frames": [900, 1100, 1500, 2100], + "traces": [{"n": "Screen Interactive", "ph": "s", "t": 1050}, {"n": "Screen Interactive", "ph": "e", "t": 2000}], + "calls": [["event", "Screen Viewed", "viewed", 2100]] +} +``` + +`traces` and `calls` are optional. The first occurrence of each mark/span phase +is used. An end-mark list requires every named mark and takes their maximum, +not a subsequent layout caused by a later re-render. `--event` matches the call's third field and uses its fourth as the JS +timestamp. Do not mix host wall time with these monotonic timestamps. + +For existing exports, the reporter also reads `blocks.txt` lines such as +`a1 baseline status=pass`, with `--rounds` supplied explicitly. Preserve the +alternating block order and use `--scenario` to describe its actual raw layout. diff --git a/domains/agentic/skills/recipe-performance/references/mobile-device-measurement.md b/domains/agentic/skills/recipe-performance/references/mobile-device-measurement.md new file mode 100644 index 00000000..8e061e64 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/references/mobile-device-measurement.md @@ -0,0 +1,163 @@ +# Mobile bottleneck diagnosis + +Use the task-locked harness and an owned device, iOS simulator or Android +emulator session. The same recipe, timing boundaries, CPU capture and elimination +method apply to each. Resolve the iOS simulator UDID through the runtime contract +and use the harness's simulator lifecycle/UI actions; bind Android to its exact +adb serial, including an emulator serial. Resolve the +current app, Metro endpoint, device serial and recipe commands from its runtime +contract/help. Preparation stays outside the measured window. A dev client with +Metro can compare JS changes without a native build; record its native revision, +served bundle SHA and dev mode. Do not call dev-bundle numbers release timings. + +Follow the real-device diagnosis and final-confirmation rule in the skill's +How to use section. Emulator CPU allocation, host load, +debug mode and virtual rendering can distort frame rates. Record the target, +OS/runtime, host and build mode; keep target and host allocation fixed within +each comparison. Simulator results apply to that environment and do not establish +physical-device latency or native GPU performance. + +## Verify the requested source first + +Before source attribution or a timed open, resolve the requested baseline to an +exact commit. For "main", fetch and resolve `origin/main`; a branch name or an +idle checkout is not evidence of that state. Compare it with the task checkout's +HEAD before creating the task branch. If preparation was skipped or the slot is +warm, this check is still required. Preserve unrelated work and use the approved +workspace/prepare path to select the requested commit. Verify the served bundle +after reload and retain its source receipt. If dependencies or native +compatibility prevent that state from running, report BLOCKED instead of +measuring an older release and calling it main. + +Use the product's production trace sampler during diagnosis as well as paired +proof, including custom sampler overrides. Record the effective rate and its +source. Dev builds that sample every span can add synchronous CPU work and send +the investigation toward a development-only cost. + +## Classify the delay + +Measure the UI gesture, JS handler entry, first following JS rAF callback, first +layout, and the actual ready/transition boundary separately. Use one monotonic +JS clock. JS handler timing excludes any wait before that handler executes. + +| Observation during the same action | Next evidence | +| --- | --- | +| JS rAF stops, long JS tasks, tap handler/layout delayed | JS CPU sampling and React commit attribution | +| JS continues but native frames miss deadlines | Android frame timeline/Perfetto or iOS native profiler for UI/render work | +| Both stall | Correlate both timelines; do not force a single-thread explanation | +| Idle JS/native threads while readiness waits | Request/dependency and lifecycle trace | + +Use the RN performance monitor for triage, then save repeatable evidence. JS rAF +cadence is not compositor FPS. On Android, collect package-scoped `dumpsys gfxinfo` +frame stats or Perfetto around the recipe action, with timestamps and support +limits. A screenshot, delayed timer, or total render count alone cannot classify +the bottleneck. + +## Automate a CPU profile + +Profile a separate open from the timing series. Navigate and settle first, start +Hermes/JS sampling immediately before the same recipe-driven gesture, stop after +the declared ready boundary, and retain the recipe trace alongside the profile. + +The native bridge already exposes `profiler-start` and `profiler-stop --out`. +Find the installed bridge through the task harness's help/runtime and confirm +these commands exist. They send CDP `Profiler.start`/`Profiler.stop`; Hermes may +reject `Profiler.enable` even when start/stop work. For example, in a tracked +shell job with `set -e` and the task's resolved command paths: + +```bash +# BRIDGE is the resolved mobile bridge; FLOW_RECIPE performs one measured open. +# Preparation, warm-up and navigation have already completed. +OUT= +mkdir -p "$OUT" +node "$BRIDGE" profiler-start > "$OUT/profile-start.json" +trap 'node "$BRIDGE" profiler-stop --out "$OUT/interrupted.cpuprofile"' EXIT +"$HARNESS" run "$FLOW_RECIPE" --artifacts-dir "$OUT/recipe" --json +node "$BRIDGE" profiler-stop --out "$OUT/open.cpuprofile" > "$OUT/profile-stop.json" +trap - EXIT +python3 "$PERF_SKILL_DIR/scripts/profile-flame.py" \ + "$OUT/open.cpuprofile" "$OUT/flame.html" +``` + +Bind the bridge to the same slot endpoint as the harness using its documented +runtime options. Do not guess a default port. If its wrapper lacks these +commands, use one CDP connection to the exact Metro `/json/list` app target: +`Profiler.start`, execute the recipe, `Profiler.stop`, save `result.profile`. +Always stop sampling on failure; retain the failure rather than presenting a +partial run as a success. + +Open `flame.html` and inspect its sample-weighted caller graph and self-time +ranking. Also import `.cpuprofile` into RN DevTools if source maps are needed. +Record sample count, sample gaps, window length and whether the slow action is +actually represented. Zero samples, root/idle-only samples, or a sampler starved +during the stall are missing CPU attribution. Keep that evidence and diagnose +the profiler path; a React trace or wrapper timer does not become a CPU profile. +Correlate sampled functions with current source and React commits before naming +a component as the cause. Profiling overhead belongs outside the paired proof. + +## Keep inspector messages small + +Large CDP replies can stall the app themselves. In a prior Android investigation, +a full fiber-tree reply never completed, left a zero-byte output and caused an +ANR. The Android main-thread stack was in inspector `send`, large-object +allocation and GC. A reduced collect of about 125 KB completed; about 23 MB of +trace data delivered as many `Tracing.dataCollected` events also completed. +The failed single-message size was not measured. These observations establish +neither a universal safe cap nor the exact failure threshold. + +Start with a short action-only sampling window. Read small summaries instead of +serializing an entire fiber tree. For custom probes, keep bounded records on the +runtime and retrieve bounded slices. Where the installed runtime supports native +sampling to a device file, stop there and `adb pull` the file. Chunked tracing +also avoids one giant reply. `Profiler.stop` itself may return one large message; +splitting its JSON on the host after receipt cannot prevent that stall. + +Record actual serialized bytes per message or chunk, total bytes, elapsed +transfer time and the largest successful payload. If a transfer fails, record +the attempted size only if known, its timeout/error, app responsiveness and +package-scoped ANR stack. State the threshold as an observed bound, or unknown. +Do not repeatedly grow payloads until the phone ANRs. Shorten the window or use +a supported file/stream path before repeating the user interaction. + +## Eliminate causes + +Use reversible probes and retain each diagnostic patch. Before choosing a fix, +measure how much of the delay comes from mounting the content: compare the normal +flow with a temporary blank shell that preserves the route, gesture, transition +and ready observations. Put the switch above the tested subtree so its hooks do +not run while omitted. Returning null after those hooks ran does not remove their +mount cost; keep the surrounding component's hook order stable. + +Add sections back progressively, or halve the expensive subtree. If omitting +content barely helps, investigate work before navigation, surrounding providers +or native rendering instead. Runtime/library frames still count when this +screen causes them; attribute them to the app's mounted work. If a candidate +explains only a small part of a large JS gap, continue this coarse elimination +before settling on that micro-optimization. Choose the largest proven removable +cost, not the first small positive result. Keep fixture and state unchanged. + +Read runtime switches at mount so one session can rotate conditions between +opens. Record each condition, first JS frame, readiness, long gaps and relevant +CPU/React evidence. Confirm with an on/off/on reversal. A missing view that moves +work elsewhere is an attribution experiment, not a shippable fix. Do not hide +errors, remove required functionality, or claim several switches prove each +one's contribution. A valid fix may reduce work or defer it past first response; +retain all controls and measure full readiness and deferred work separately. +Trace the causal chain from gesture through scheduled work +and layout/transition, including hidden mounted children and pre-existing screen +work when the evidence points there. + +Propose the smallest fix to the proven chain. For an authorised fix, prove the code +states with [paired device measurement](paired-device-measurement.md), including +adjacent-screen/deferred-work checks. For a requested PR comparison, use that +reference directly; no new fix or approval cycle is needed. + +## Existing guides + +Use the installed `mms-performance/references/` copies when available, or these +source guides. Keep their detailed tool instructions in one place: + +- [React/JS profiling](https://github.com/MetaMask/skills/blob/main/domains/performance/skills/performance/references/js-profile-react.md) +- [JS and UI frame rates](https://github.com/MetaMask/skills/blob/main/domains/performance/skills/performance/references/js-measure-fps.md) +- [Performance timeline](https://github.com/MetaMask/skills/blob/main/domains/performance/skills/performance/references/js-performance-panel.md) +- [MetaMask tools](https://github.com/MetaMask/skills/blob/main/domains/performance/skills/performance/references/mm-tools.md) diff --git a/domains/agentic/skills/recipe-performance/references/paired-device-measurement.md b/domains/agentic/skills/recipe-performance/references/paired-device-measurement.md new file mode 100644 index 00000000..610cd476 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/references/paired-device-measurement.md @@ -0,0 +1,105 @@ +# Paired device proof + +Read this for a requested comparison or after an approved fix. Profiling and +elimination opens are separate from these unprofiled measurement blocks. + +## Workflow + +1. Freeze the device/simulator/emulator, scenario, source refs, installed dependencies, build mode, + production tracing configuration, warm-up, settle time and timed-open count. + Use the same dev client and Metro settings in both states. Dev-bundle times + are relative comparisons, not release-build timings. If lockfiles differ, + state which shared dependency tree was used; do not call it an unmodified + shipping-build comparison. Use an owned measurement checkout/device session. For an authorised + uncommitted fix, freeze and retain the candidate patch and its hash over the + baseline ref; the adapter applies it only in candidate blocks and records + that delivered state. Do not label an uncommitted patch a shipping commit. +2. Prepare a screen-specific capture adapter using the [adapter contract](references/capture-adapter.md). + It checks out the requested source, applies identical diagnostic marks, + restarts the app, verifies the served source and effective trace sampling, + settles, performs one unrecorded warm-up, then captures the planned opens. + Use a fixed settle interval and the application's actual production trace + sampler, including custom sampler overrides. Read back the effective rate; + dev sampling can add synchronous work that disappears in production. Keep device, + fixture/account state, network, recorder and HUD settings equal across states. +3. Verify one capture before repeating: correct route/control, unobscured target, + sampling readback, usable marks and a single chronological rAF sample chain. + A shared fixture may change the action between blocks. Invalid setup is not + a slow open. Freeze recipes, probes and driver scripts before the series; + only planned state/condition switches occur between blocks. Finish source, + test, instrumentation, recipe and driver edits before starting the series. + The harness fingerprint includes test files: adding a test during a capture + invalidates the series even when it is not bundled into the app. Finish + validation edits first; run CPU-heavy repository checks outside timing runs. +4. Alternate baseline/candidate blocks with an app restart per block. Two blocks + per state and five timed opens per block give n=10 per state per scenario. + Keep each warm-up out of the timed files. Run as a tracked job and inspect + every block's exit status. A failed block stops the driver and invalidates + the comparison; retain it, fix the recorded setup cause and use a fresh output + directory. Never discard a block because its numbers are unwelcome. + +```bash +PERF_SKILL_DIR= +BASE_SHA=$(git rev-parse 'origin/main^{commit}') +HEAD_SHA=$(git rev-parse 'HEAD^{commit}') +OUT=temp/mobile-perf/paired-01 + +# The executable adapter is prepared for this screen, not supplied by the kit. +python3 "$PERF_SKILL_DIR/scripts/paired-blocks.py" "$OUT" 5 2 \ + "baseline:$BASE_SHA" "candidate:$HEAD_SHA" ./capture-block.sh +python3 "$PERF_SKILL_DIR/scripts/report.py" "$OUT" baseline candidate \ + --scenario 'open=data/timed-*.json' --json "$OUT/metrics.json" > "$OUT/report.md" +``` + +Resolve `PERF_SKILL_DIR` to this skill's installed directory if the operator uses a +different skill root. The driver only invokes the adapter; it never resets a +checkout. Its manifest records **requested** refs, not proof of delivered code. + +## Timing definitions + +Use one monotonic JS clock for marks, rAF timestamps, spans and events. Start is +the `press` handler mark. End is `open-complete`: for a sheet, the later of slide +completion and content layout; for navigation, the declared ready/transition +boundary. These JS timings exclude delay before the handler and do not prove +native input-to-paint latency. rAF callback FPS is JS cadence, not compositor FPS. + +| Metric | Definition | +| --- | --- | +| window ms | end minus press | +| fps | rAF callbacks inside the inclusive window divided by its seconds | +| next JS frame ms | press to the first subsequent JS rAF callback | +| >100 ms frames | whole JS frame gaps over 100 ms that overlap the window | +| longest frame | longest whole overlapping JS gap | +| blocked ms | sum of `max(0, gap - 50 ms)` over whole overlapping gaps | + +Whole-gap metrics include the portions outside the window at its boundaries; +blocked ms can exceed open ms. This preserves the reference method. Do not mix +them with clipped-window or native jank metrics. Missing end/frame data fails +analysis rather than falling back to a fixed-duration window. + +Use `--start-mark` / `--end-mark` for another declared interval. For separate +slide/layout marks, use `--end-mark 'slide-end,content-layout'`: both are required +and the later first occurrence ends the window. Analyze scenarios with different +completion contracts separately. Optional +`--layout-mark`, `--span 'label=actual span name'` and +`--event 'label=actual event key'` retain first layout, product spans and event +timing relative to the end. Event timing has no automatic better/worse direction. +Multiple variants use repeated `--scenario 'name=relative/timed-*.json'` options. + +## Report and decide + +- Keep raw captures, block/source/build/sampler metadata, n and p50/p75. The + reporter uses median and linearly interpolated p75 over individual opens, + and retains each block's median separately. Missing optional spans show their + actual n; they never become zero. +- `better-separated` means **every** candidate block median beats **every** + baseline block median, with FPS higher and durations/gaps lower. The reverse + is `worse-separated`; overlap/ties are `not-separated`. This is a conservative + drift check, not a significance test or confidence interval. A pooled gain + without block separation stays inconclusive at this sample size. +- Report tail regressions even when p50 improves. Measure the adjacent screen + and a declared post-navigation interval when work may have moved there. + First response, content readiness and deferred work are separate quantities. + Combined changes do not establish any individual fix's contribution. +- Keep traces and unattributed time separate from measured latency. A starved + CPU sampler is missing evidence, not proof that no CPU work happened. diff --git a/domains/agentic/skills/recipe-performance/references/performance-evidence.md b/domains/agentic/skills/recipe-performance/references/performance-evidence.md new file mode 100644 index 00000000..7e793627 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/references/performance-evidence.md @@ -0,0 +1,61 @@ +# Performance evidence + +## Choose the authority + +| Question | Primary evidence | Common mistake | +|---|---|---| +| Did the real user journey execute? | Recipe UI action + state assertion | Calling the controller instead of the UI | +| Did the product CUF start/end correctly? | Fresh, correlated product markers or spans | Treating recipe node duration as CUF latency | +| Is indexed telemetry correct? | Exact Sentry span/query in the run window | Broadening the query until rows appear | +| Why is Extension slow? | DevTools Performance trace over the same action | Adding a harness wrapper before direct CDP is tried | +| Did state converge? | Controller read followed by teardown assertion | Inferring state from a screenshot | + +## Lifecycle semantics + +- `cold`: terminate and relaunch the installed app; prove readiness before the measured action. +- `warm`: keep the process and relevant runtime initialized. +- `background_resume`: background and foreground the same running process. A foreground action that reloads JavaScript is not valid background-resume evidence. + +Record the observed lifecycle context at the product boundary. Do not infer it from the recipe mode or elapsed time. + +## Secrets + +Use Recipe Harness's fixture-selection flow through the task-locked CLI. If the fixture is missing or invalid, relay doctor's exact choices and wait for the operator. Reuse an approved canonical fixture when offered; do not assume a new wallet must be initialized. Only a human operator supplies missing SRPs and passwords. + +The team 1Password vault (for human operators only) is: +https://consensys.1password.com/app#/m3mshk44tttveurtljcuwdj46m/AllItems + +Never request or paste an SRP, private key, or password in chat or CLI arguments. Never hard-code these values in recipes, scripts, or evidence artifacts. Agents must not access the vault or any secret store directly. + +## Sentry + +Confirm the build can emit: DSN/config present, test overrides off, consent enabled where required, SDK initialized, and the environment sampled. Query the exact current span/transaction name and bounded run window. A zero-row query is `INCONCLUSIVE` until local emission and ingestion prerequisites are proven. + +Prefer a scoped read token. If a human-authenticated browser is explicitly provided, keep access inside that browser session and retain only redacted response fields. Never extract cookies, storage, auth headers, session tokens, wallet secrets, or recovery phrases. + +## Extension DevTools and CDP + +Use `mm-harness debug` for a human Performance-panel recording attached to the slot-owned Extension page. Record only the action window and save the trace as an artifact. + +For automated diagnosis, read the actual CDP endpoint from `mm-harness status --json` or the checkout runtime contract—never assume a port. Select the exact Extension target from `/json/list`, then attach to its reported `webSocketDebuggerUrl`. + +- `Performance.enable` + `Performance.getMetrics` provides cumulative runtime counters. Capture before and after the same action and compare deltas. +- `Tracing.start` + `Tracing.end` provides event-level CPU/render/network evidence. Wait for `Tracing.tracingComplete` and persist the complete stream before analyzing it. + +The DevTools protocol can change between Chromium versions. Use the target browser's `/json/protocol` contract when automating beyond these stable calls. These Chrome domains do not establish Mobile React Native performance; Mobile CUFs require product markers/spans and real lifecycle recipes. + +## Cross-client claims + +- Mobile: real UI, lifecycle, product CUF markers, Sentry, controller end state. +- Extension: real UI and Chrome trace; use product telemetry only when an equivalent boundary exists in current source. +- Core: controller/state and headless operation timing only; no UI render or lifecycle claim. + +Reuse the journey across clients, but state each client's proof claim separately. + +### Mobile: obtaining a build + +A local build is always valid but takes significantly longer. As a faster alternative, download a pre-built arm64 APK from the scheduled CI workflow: + +https://github.com/MetaMask/metamask-mobile/actions/workflows/build-android-arm64-scheduled.yml + +Pick the latest successful run, download the arm64 APK artifact, and install it on the target device or emulator before running the recipe. For native changes, confirm the build SHA matches the source revision under investigation before recording evidence. For a JS-only dev-client comparison, keep the same compatible native client in both states and verify each served Metro bundle revision separately. Record both identities; do not claim release-build timings from a dev bundle. diff --git a/domains/agentic/skills/recipe-performance/references/rendering-network.md b/domains/agentic/skills/recipe-performance/references/rendering-network.md new file mode 100644 index 00000000..c9fc16fe --- /dev/null +++ b/domains/agentic/skills/recipe-performance/references/rendering-network.md @@ -0,0 +1,50 @@ +# Extension rendering and network optimization + +Use a deterministic recipe to create the measured action. Keep profiler collection outside the recipe protocol unless repeated use proves a generic action is needed. + +## Capture contract + +1. Name one user action and its start/end observation. +2. Reuse the closest working recipe; move preparation, fixture convergence, and navigation before the measured window. +3. Resolve the slot-owned Extension target and CDP endpoint. Confirm available commands from that browser's `/json/protocol`; never assume a port or protocol version. +4. Capture only the action window: + - `Network.enable` and request/response/finish events; + - `Performance.enable` plus before/after `Performance.getMetrics` deltas; + - `Tracing.start`/`Tracing.end`, retaining events through `Tracing.tracingComplete`. +5. Preserve normal cache/service-worker behavior unless the claim explicitly concerns cold-cache behavior. Record any override. +6. Save the raw trace and a redacted summary beside the recipe artifacts. + +`mm-harness debug` is the human path to the slot-owned DevTools Performance panel. Direct CDP is the agent path. Neither requires a new recipe action. For a repeatable agent capture around one explicit command, run: + +```bash +.agents/skills/mms-recipe-performance/scripts/capture-extension-window.mjs \ + --boundary '' \ + --out-dir /artifacts/baseline -- \ + mm-harness call +``` + +The helper resolves the CDP port from the checkout's runtime contract (an explicit `--cdp-port`, `RECIPE_CDP_PORT`, or `CDP_PORT` overrides it), then retains redacted CDP events and a summary. The caller supplies domain routes, selectors, and action parameters from its team library or live discovery. + +## Analyze + +### Network + +Report total requests, transferred bytes, duration, initiator, and dependency ordering for the measured window. Group duplicate candidates by method plus normalized URL and, when safe, a non-secret body digest. Strip query values, headers, bodies, credentials, and wallet data from retained evidence. + +A repeated request is not automatically a bug. Classify retry, polling, cache validation, batching, and distinct request bodies before declaring duplication. Prove an avoidable duplicate from its initiator and current source. + +Rank slow or blocking requests by their effect on the claimed UI boundary, not duration alone. A long background request that does not gate rendering is a side finding. + +### Main thread and rendering + +Report long tasks and the measured window's scripting, style, layout, paint, and composite work. Use trace events and screenshots to connect expensive work to the claimed screen; do not infer user impact from a global process total. + +### React + +Use a verified React Profiler channel to report commit count, commit duration, and repeatedly expensive components. DOM mutations, paint events, and function-call counts do not prove a React rerender. If the running build exposes no verified React profiler channel, record React commit analysis as `N/A` and retain the browser trace. + +## Prove the optimization + +Treat duplicate requests, long tasks, and expensive commits as hypotheses until source and trace evidence identify a causal path. Apply one minimal fix, then rerun the identical recipe with the same lifecycle, fixture, cache policy, target surface, and profiler settings. + +Compare before/after request inventory, blocking chain, main-thread/rendering breakdown, React commits when available, product boundary markers, and final controller state. If normal run variance can reverse the result, repeat the unchanged pair or report `INCONCLUSIVE`; never select only the fastest run. diff --git a/domains/agentic/skills/recipe-performance/scripts/capture-extension-window.mjs b/domains/agentic/skills/recipe-performance/scripts/capture-extension-window.mjs new file mode 100755 index 00000000..f4ce3114 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/capture-extension-window.mjs @@ -0,0 +1,453 @@ +#!/usr/bin/env node + +import { readFileSync } from 'node:fs'; +import { mkdir, writeFile } from 'node:fs/promises'; +import { spawn } from 'node:child_process'; +import path from 'node:path'; +import { pathToFileURL } from 'node:url'; + +function usage(message) { + if (message) console.error(`capture-extension-window: ${message}`); + console.error('Usage: capture-extension-window.mjs [--cdp-port ] --boundary --out-dir [--timeout-ms ] -- [args...]'); + process.exit(message ? 2 : 0); +} + +function parseArgs(argv) { + if (argv.includes('-h') || argv.includes('--help')) usage(); + const split = argv.indexOf('--'); + if (split < 0 || split === argv.length - 1) usage('pass the measured command after --'); + const options = argv.slice(0, split); + const command = argv.slice(split + 1); + const parsed = { timeoutMs: 15_000, command }; + for (let index = 0; index < options.length; index += 1) { + const key = options[index]; + const value = options[index + 1]; + if (!value) usage(`missing value for ${key}`); + if (key === '--cdp-port') parsed.cdpPort = Number(value); + else if (key === '--boundary') parsed.boundary = value; + else if (key === '--out-dir') parsed.outDir = value; + else if (key === '--timeout-ms') parsed.timeoutMs = Number(value); + else usage(`unknown option ${key}`); + index += 1; + } + if (parsed.cdpPort !== undefined && (!Number.isInteger(parsed.cdpPort) || parsed.cdpPort < 1)) usage('--cdp-port must be a positive integer'); + if (!parsed.boundary) usage('--boundary is required'); + if (!parsed.outDir) usage('--out-dir is required'); + if (!Number.isFinite(parsed.timeoutMs) || parsed.timeoutMs < 1) usage('--timeout-ms must be positive'); + return parsed; +} + +function resolveCdpPort(explicitPort) { + const overrides = [ + ['--cdp-port', explicitPort], + ['RECIPE_CDP_PORT', process.env.RECIPE_CDP_PORT], + ['CDP_PORT', process.env.CDP_PORT], + ]; + const override = overrides.find(([, value]) => value !== undefined && value !== ''); + if (override) { + const [source, rawPort] = override; + const cdpPort = Number(rawPort); + if (!Number.isInteger(cdpPort) || cdpPort < 1) { + throw new Error(`${source} must be a positive integer`); + } + return { cdpPort, source }; + } + + const runtimePath = path.resolve('temp/recipe/runtime/agentic-runtime.json'); + try { + const runtime = JSON.parse(readFileSync(runtimePath, 'utf8')); + const cdpPort = Number(runtime.cdpPort); + if (Number.isInteger(cdpPort) && cdpPort > 0) return { cdpPort, source: runtimePath }; + } catch (error) { + if (error?.code !== 'ENOENT') throw new Error(`could not read ${runtimePath}: ${error.message}`); + } + throw new Error('no authoritative Extension CDP port found; run mm-harness doctor --fix or pass --cdp-port '); +} + +class CdpClient { + constructor(socket) { + this.socket = socket; + this.nextId = 0; + this.pending = new Map(); + this.listeners = new Map(); + socket.onmessage = ({ data }) => this.receive(JSON.parse(String(data))); + } + + static async connect(url) { + if (typeof WebSocket !== 'function') throw new Error('Node WebSocket support is required'); + const socket = new WebSocket(url); + await new Promise((resolve, reject) => { + socket.onopen = resolve; + socket.onerror = () => reject(new Error(`could not connect to ${url}`)); + }); + return new CdpClient(socket); + } + + receive(message) { + if (message.id && this.pending.has(message.id)) { + const pending = this.pending.get(message.id); + this.pending.delete(message.id); + if (message.error) pending.reject(new Error(`${pending.method}: ${message.error.message}`)); + else pending.resolve(message.result ?? {}); + return; + } + if (!message.method) return; + for (const listener of this.listeners.get(message.method) ?? []) listener(message.params ?? {}); + } + + send(method, params = {}) { + const id = ++this.nextId; + return new Promise((resolve, reject) => { + this.pending.set(id, { method, resolve, reject }); + this.socket.send(JSON.stringify({ id, method, params })); + }); + } + + on(method, listener) { + const listeners = this.listeners.get(method) ?? []; + listeners.push(listener); + this.listeners.set(method, listeners); + } + + once(method, timeoutMs) { + return new Promise((resolve, reject) => { + const remove = (listener) => { + const listeners = this.listeners.get(method) ?? []; + this.listeners.set(method, listeners.filter((candidate) => candidate !== listener)); + }; + const listener = (params) => { + clearTimeout(timer); + remove(listener); + resolve(params); + }; + const timer = setTimeout(() => { + remove(listener); + reject(new Error(`timed out waiting for ${method}`)); + }, timeoutMs); + this.on(method, listener); + }); + } + + close() { + this.socket.close(); + } +} + +async function extensionTarget(port) { + const response = await fetch(`http://127.0.0.1:${port}/json/list`); + if (!response.ok) throw new Error(`CDP target list returned HTTP ${response.status}`); + const targets = await response.json(); + const matches = targets.filter((target) => target.type === 'page' && /^chrome-extension:\/\/.+\/home\.html/.test(target.url)); + if (matches.length !== 1) throw new Error(`expected one MetaMask home.html page target on CDP :${port}, found ${matches.length}`); + if (!matches[0].webSocketDebuggerUrl) throw new Error('selected Extension target has no webSocketDebuggerUrl'); + return matches[0]; +} + +export function run(command, timeoutMs) { + return new Promise((resolve, reject) => { + const detached = process.platform !== 'win32'; + const child = spawn(command[0], command.slice(1), { + cwd: process.cwd(), + env: process.env, + stdio: 'inherit', + detached, + }); + let timedOut = false; + let interruptedBy; + let forceKillTimer; + let timeout; + let cleanupPromise = Promise.resolve(); + const kill = (signal) => { + if (!child.pid) return; + try { + if (detached) process.kill(-child.pid, signal); + else child.kill(signal); + } catch (error) { + if (error?.code !== 'ESRCH') throw error; + } + }; + const waitForGroupExit = async () => { + if (!detached || !child.pid) return; + for (let attempt = 0; attempt < 100; attempt += 1) { + try { + process.kill(-child.pid, 0); + } catch (error) { + if (error?.code === 'ESRCH') return; + throw error; + } + await new Promise((resolve) => setTimeout(resolve, 10)); + } + throw new Error('measured command process group did not terminate'); + }; + const scheduleForceKill = (delayMs) => { + cleanupPromise = new Promise((resolve, rejectCleanup) => { + forceKillTimer = setTimeout(async () => { + try { + kill('SIGKILL'); + await waitForGroupExit(); + resolve(); + } catch (error) { + rejectCleanup(error); + } + }, delayMs); + }); + }; + const signalHandlers = new Map(['SIGHUP', 'SIGINT', 'SIGQUIT', 'SIGTERM'].map((signal) => [ + signal, + () => { + if (interruptedBy) return; + interruptedBy = signal; + clearTimeout(timeout); + kill(signal); + scheduleForceKill(1_000); + }, + ])); + const removeSignalHandlers = () => { + for (const [signal, handler] of signalHandlers) process.off(signal, handler); + }; + for (const [signal, handler] of signalHandlers) process.on(signal, handler); + timeout = setTimeout(() => { + timedOut = true; + kill('SIGTERM'); + scheduleForceKill(Math.min(timeoutMs, 1_000)); + }, timeoutMs); + child.once('error', (error) => { + clearTimeout(timeout); + clearTimeout(forceKillTimer); + removeSignalHandlers(); + reject(error); + }); + child.once('exit', async (code, signal) => { + clearTimeout(timeout); + if (!timedOut && !interruptedBy) clearTimeout(forceKillTimer); + removeSignalHandlers(); + try { + if (timedOut || interruptedBy) await cleanupPromise; + } catch (error) { + reject(error); + return; + } + if (timedOut) { + reject(new Error(`measured command timed out after ${timeoutMs}ms`)); + return; + } + if (interruptedBy) { + const error = new Error(`measured command interrupted by ${interruptedBy}`); + error.signal = interruptedBy; + reject(error); + return; + } + if (code === 0) resolve(); + else reject(new Error(`measured command failed (${signal ?? `exit ${code ?? 'unknown'}`})`)); + }); + }); +} + +export function terminateWithSignal(signal) { + process.kill(process.pid, signal); +} + +async function waitForBoundary(client, selector, timeoutMs) { + const deadline = Date.now() + timeoutMs; + const expression = `(() => { const node = document.querySelector(${JSON.stringify(selector)}); if (!node) return false; const style = getComputedStyle(node); const rect = node.getBoundingClientRect(); return style.display !== 'none' && style.visibility !== 'hidden' && rect.width > 0 && rect.height > 0; })()`; + while (Date.now() < deadline) { + const result = await client.send('Runtime.evaluate', { expression, returnByValue: true }); + if (result.result?.value === true) return; + await new Promise((resolve) => setTimeout(resolve, 50)); + } + throw new Error(`boundary did not become visible within ${timeoutMs}ms: ${selector}`); +} + +function redactPlainText(value) { + return String(value) + .replace(/\bBearer\s+\S+/giu, 'Bearer [REDACTED]') + .replace(/\b(password|passwd|token|secret|mnemonic|seed|private[_-]?key)\s*[:=]\s*([^\s,;&]+)/giu, '$1=[REDACTED]') + .replace(/0x[a-fA-F0-9]{16,}/g, '[REDACTED_HEX]'); +} + +function sensitivePathSegment(segment) { + let decoded = segment; + try { + decoded = decodeURIComponent(segment); + } catch {} + return /^0x[a-f0-9]{16,}$/iu.test(decoded) + || decoded.length >= 24; +} + +export function normalizeUrl(value) { + try { + const url = new URL(value); + url.username = ''; + url.password = ''; + for (const key of [...url.searchParams.keys()]) url.searchParams.set(key, ''); + url.hash = ''; + url.pathname = url.pathname + .split('/') + .map((segment) => (sensitivePathSegment(segment) ? encodeURIComponent('[REDACTED]') : segment)) + .join('/'); + return redactPlainText(url.toString()); + } catch { + return redactPlainText(value); + } +} + +function sanitizeString(value) { + const withSafeUrls = String(value).replace( + /\b(?:https?|wss?):\/\/[^\s"'<>]+/giu, + (url) => normalizeUrl(url), + ); + return redactPlainText(withSafeUrls); +} + +export function sanitize(value, key = '') { + if (value === null || value === undefined) return value; + if (/headers|postData|password|mnemonic|private|secret|seed|vault|token|authorization|cookie|session|credential|api[-_]?key/i.test(key)) return ''; + if (typeof value === 'string') return /url/i.test(key) ? normalizeUrl(value) : sanitizeString(value); + if (Array.isArray(value)) return value.map((item) => sanitize(item)); + if (typeof value === 'object') return Object.fromEntries(Object.entries(value).map(([childKey, childValue]) => [childKey, sanitize(childValue, childKey)])); + return value; +} + +function metricsMap(result) { + return Object.fromEntries((result.metrics ?? []).map(({ name, value }) => [name, value])); +} + +function metricDeltas(before, after) { + const names = ['TaskDuration', 'ScriptDuration', 'LayoutDuration', 'RecalcStyleDuration', 'LayoutCount', 'RecalcStyleCount']; + return Object.fromEntries(names.map((name) => [name, Number(((after[name] ?? 0) - (before[name] ?? 0)).toFixed(6))])); +} + +function networkSummary(events) { + const requests = new Map(); + for (const event of events) { + const id = event.params.requestId; + const record = requests.get(id) ?? { requestId: id }; + if (event.method === 'Network.requestWillBeSent') { + record.method = event.params.request?.method; + record.url = normalizeUrl(event.params.request?.url ?? ''); + record.initiator = sanitize(event.params.initiator ?? null); + record.startedAt = event.params.timestamp ?? null; + } else if (event.method === 'Network.responseReceived') { + record.status = event.params.response?.status ?? null; + record.protocol = event.params.response?.protocol ?? null; + } else if (event.method === 'Network.loadingFinished') { + record.encodedDataLength = event.params.encodedDataLength ?? 0; + record.finishedAt = event.params.timestamp ?? null; + } else if (event.method === 'Network.loadingFailed') { + record.failed = true; + record.errorText = sanitizeString(event.params.errorText ?? 'unknown'); + } + requests.set(id, record); + } + const rows = [...requests.values()].filter((row) => row.url); + const groups = new Map(); + for (const row of rows) { + const key = `${row.method ?? 'GET'} ${row.url}`; + const group = groups.get(key) ?? []; + group.push(row.requestId); + groups.set(key, group); + } + return { + requestCount: rows.length, + transferredBytes: rows.reduce((sum, row) => sum + (row.encodedDataLength ?? 0), 0), + duplicateCandidates: [...groups.entries()].filter(([, ids]) => ids.length > 1).map(([key, requestIds]) => ({ key, count: requestIds.length, requestIds })), + requests: rows, + }; +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const resolvedPort = resolveCdpPort(options.cdpPort); + options.cdpPort = resolvedPort.cdpPort; + const outDir = path.resolve(options.outDir); + await mkdir(outDir, { recursive: true }); + const target = await extensionTarget(options.cdpPort); + const client = await CdpClient.connect(target.webSocketDebuggerUrl); + const networkEvents = []; + const traceEvents = []; + let before = {}; + let after = {}; + let startedAt = null; + let finishedAt = null; + let failure = null; + let interruptedSignal = null; + let tracingStarted = false; + + for (const method of ['Network.requestWillBeSent', 'Network.responseReceived', 'Network.loadingFinished', 'Network.loadingFailed']) { + client.on(method, (params) => networkEvents.push({ method, params })); + } + client.on('Tracing.dataCollected', ({ value }) => traceEvents.push(...(value ?? []))); + + try { + await client.send('Network.enable'); + await client.send('Performance.enable'); + before = metricsMap(await client.send('Performance.getMetrics')); + await client.send('Tracing.start', { + categories: 'devtools.timeline,v8.execute,blink.user_timing,loading', + transferMode: 'ReportEvents', + }); + tracingStarted = true; + startedAt = new Date().toISOString(); + await run(options.command, options.timeoutMs); + await waitForBoundary(client, options.boundary, options.timeoutMs); + finishedAt = new Date().toISOString(); + after = metricsMap(await client.send('Performance.getMetrics')); + } catch (error) { + failure = sanitizeString(error instanceof Error ? error.message : String(error)); + if (typeof error?.signal === 'string') interruptedSignal = error.signal; + } finally { + if (tracingStarted) { + try { + const complete = client.once('Tracing.tracingComplete', options.timeoutMs); + await client.send('Tracing.end'); + await complete; + } catch (error) { + failure ??= sanitizeString(error instanceof Error ? error.message : String(error)); + } + } + } + + const network = networkSummary(networkEvents); + const summary = { + schemaVersion: 1, + status: failure ? 'fail' : 'pass', + target: { id: target.id, url: normalizeUrl(target.url), cdpPort: options.cdpPort, cdpPortSource: resolvedPort.source }, + boundary: sanitizeString(options.boundary), + startedAt, + finishedAt, + durationMs: startedAt && finishedAt ? Date.parse(finishedAt) - Date.parse(startedAt) : null, + metrics: failure || Object.keys(before).length === 0 || Object.keys(after).length === 0 + ? null + : metricDeltas(before, after), + requestCount: network.requestCount, + transferredBytes: network.transferredBytes, + duplicateCandidates: network.duplicateCandidates, + reactCommits: 'N/A — no verified React Profiler channel', + ...(failure ? { error: failure } : {}), + }; + const evidence = { + ...summary, + performance: { before, after }, + network: network.requests, + traceEvents: sanitize(traceEvents), + }; + const summaryPath = path.join(outDir, 'performance-window.summary.json'); + const evidencePath = path.join(outDir, 'performance-window.events.json'); + await writeFile(summaryPath, `${JSON.stringify(summary, null, 2)}\n`); + await writeFile(evidencePath, `${JSON.stringify(evidence, null, 2)}\n`); + client.close(); + console.log(`performance summary: ${summaryPath}`); + console.log(`performance events: ${evidencePath}`); + if (interruptedSignal) { + terminateWithSignal(interruptedSignal); + return; + } + if (failure) throw new Error(failure); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + main().catch((error) => { + console.error(`capture-extension-window: ${sanitizeString(error instanceof Error ? error.message : String(error))}`); + process.exitCode = 1; + }); +} diff --git a/domains/agentic/skills/recipe-performance/scripts/paired-blocks.py b/domains/agentic/skills/recipe-performance/scripts/paired-blocks.py new file mode 100644 index 00000000..b39b27fa --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/paired-blocks.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Run alternating capture blocks; the caller supplies the screen adapter.""" +import argparse +import json +from pathlib import Path +import subprocess +import sys + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('out', type=Path) + parser.add_argument('rounds', type=int, help='timed opens per scenario per block') + parser.add_argument('repeats', type=int, help='blocks per state, at least two') + parser.add_argument('baseline', help='label:commit') + parser.add_argument('candidate', help='label:commit') + parser.add_argument('capture', type=Path, help='executable screen capture adapter') + args = parser.parse_args() + states = [value.split(':', 1) for value in (args.baseline, args.candidate)] + if (args.rounds < 1 or args.repeats < 2 or any(len(s) != 2 or not all(s) for s in states) + or states[0][0] == states[1][0]): + parser.error('use positive rounds, at least two repeats and distinct label:commit states') + capture = str(args.capture.resolve(strict=True)) + args.out.mkdir(parents=True, exist_ok=False) + manifest = {'rounds': args.rounds, 'repeats': args.repeats, 'states': states, 'blocks': []} + for repeat in range(1, args.repeats + 1): + for arm, (state, ref) in enumerate(states): + block = f'{"ab"[arm]}{repeat}' + directory = (args.out / block).resolve() + directory.mkdir() + with (directory / 'capture.log').open('w') as log: + result = subprocess.run([capture, state, ref, str(directory), str(args.rounds)], + stdout=log, stderr=subprocess.STDOUT) + row = {'block': block, 'state': state, 'requestedRef': ref, + 'status': 'pass' if result.returncode == 0 else 'failed', + 'exitCode': result.returncode} + manifest['blocks'].append(row) + (args.out / 'blocks.json').write_text(json.dumps(manifest, indent=2) + '\n') + print(f'{block} {state} status={row["status"]}', flush=True) + if result.returncode: + return result.returncode + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/domains/agentic/skills/recipe-performance/scripts/profile-flame.py b/domains/agentic/skills/recipe-performance/scripts/profile-flame.py new file mode 100755 index 00000000..0f502089 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/profile-flame.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""Render a CDP CPU profile as a sample-weighted caller graph and self ranking.""" +import argparse +from collections import defaultdict +from html import escape +import json +import math +from pathlib import Path + + +def render(profile): + nodes = {node['id']: node for node in profile['nodes']} + samples, deltas = profile.get('samples', []), profile.get('timeDeltas', []) + if not samples or len(samples) != len(deltas): + raise ValueError('CPU samples and matching timeDeltas are required') + if any(not isinstance(d, (int, float)) or not math.isfinite(d) or d < 0 for d in deltas): + raise ValueError('invalid sample time delta') + parents = {} + for node in nodes.values(): + for child in node.get('children', []): + if child not in nodes or child in parents: + raise ValueError('invalid CPU call tree') + parents[child] = node['id'] + own, total = defaultdict(float), defaultdict(float) + for sample, delta in zip(samples, deltas): + if sample not in nodes: + raise ValueError('sample references an unknown CPU node') + own[sample] += delta + cursor, seen = sample, set() + while cursor is not None: + if cursor in seen: + raise ValueError('cycle in CPU call tree') + seen.add(cursor) + total[cursor] += delta + cursor = parents.get(cursor) + weight = sum(deltas) + if weight <= 0: + raise ValueError('CPU profile has no positive sampled time') + roots = [key for key in nodes if key not in parents and total[key]] + boxes, depth = [], 0 + + def label(key): + frame = nodes[key]['callFrame'] + # Raw profiles remain local; the view needs names and locations, no URL queries. + url = frame.get('url', '').split('?', 1)[0].split('#', 1)[0].rsplit('/', 1)[-1] + return f"{frame.get('functionName') or '(anonymous)'} {url}:{frame.get('lineNumber', -1) + 1}" + + def draw(key, x, level): + nonlocal depth + pending = [(key, x, level)] + while pending: + key, x, level = pending.pop() + depth = max(depth, level) + width = total[key] / weight * 1200 + text = label(key) + title = escape(f'{text}: inclusive {total[key] / 1000:.3f} ms; self {own[key] / 1000:.3f} ms') + color = f'hsl({sum(map(ord, text)) % 55},80%,72%)' + boxes.append(f'{title}') + if width > 12: + boxes.append(f'{escape(text[:max(0, int(width / 7) - 1)])}') + boxes.append('') + child_x = x + children = [] + for child in sorted(nodes[key].get('children', []), key=lambda c: total[c], reverse=True): + if total[child]: + children.append((child, child_x, level + 1)) + child_x += total[child] / weight * 1200 + pending.extend(reversed(children)) + + x = 0 + for root in roots: + draw(root, x, 0) + x += total[root] / weight * 1200 + rows = ''.join(f'{escape(label(key))}{own[key] / 1000:.3f}' + f'{total[key] / 1000:.3f}' + for key in sorted(own, key=own.get, reverse=True) if own[key]) + window = (profile['endTime'] - profile['startTime']) / 1000 + return f'''JS CPU profile + +

JS CPU profile

{len(samples)} samples; window {window:.3f} ms; +weighted time {weight / 1000:.3f} ms; largest sample delta {max(deltas) / 1000:.3f} ms.

+

Top-down caller graph. Width is sample-weighted time, not exact function duration. +Hover for location and self/inclusive estimates. Check idle-only or missing stall coverage against the recipe timeline.

+{''.join(boxes)} +

Self-time ranking

{rows}
Function / locationSelf msInclusive ms
+''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('profile', type=Path) + parser.add_argument('html', type=Path) + args = parser.parse_args() + try: + page = render(json.loads(args.profile.read_text())) + with args.html.open('x') as output: + output.write(page) + except (ValueError, KeyError, OSError) as error: + parser.exit(1, f'profile-flame: {error}\n') + + +if __name__ == '__main__': + main() diff --git a/domains/agentic/skills/recipe-performance/scripts/report.py b/domains/agentic/skills/recipe-performance/scripts/report.py new file mode 100644 index 00000000..385fc091 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/report.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +"""Extract paired JS-window metrics, interpolated percentiles and block separation.""" +import argparse +import json +from pathlib import Path +import statistics +import sys + + +def percentile(values, fraction): + values = sorted(values) + position = (len(values) - 1) * fraction + lower = int(position) + upper = min(lower + 1, len(values) - 1) + return values[lower] + (values[upper] - values[lower]) * (position - lower) + + +def timings(path, start_mark, end_mark, layout_mark, spans, events): + data = json.loads(path.read_text()) + if isinstance(data, str): # CDP evaluators may wrap their JSON result in a JSON string. + data = json.loads(data) + marks = {} + for mark in data['marks']: + marks.setdefault(mark['n'], mark['t']) + start = marks.get(start_mark) + ends = [marks.get(name) for name in end_mark.split(',')] + if start is None or any(end is None for end in ends): + raise ValueError(f'{path}: missing or invalid {start_mark}/{end_mark} window') + end = max(ends) + if end <= start: + raise ValueError(f'{path}: missing or invalid {start_mark}/{end_mark} window') + frames = data['frames'] + if any(b < a for a, b in zip(frames, frames[1:])): + raise ValueError(f'{path}: frame timestamps are not chronological') + gaps = [b - a for a, b in zip(frames, frames[1:]) if b > start and a < end] + after = [frame for frame in frames if frame > start] + if not gaps or not after or frames[0] > start or frames[-1] < end: + raise ValueError(f'{path}: frame probe did not cover the open') + result = {'window ms': end - start, + 'fps': sum(start <= frame <= end for frame in frames) / ((end - start) / 1000), + '>100 ms frames': sum(gap > 100 for gap in gaps), + 'longest frame': max(gaps), + 'blocked ms': sum(max(0, gap - 50) for gap in gaps), + 'next JS frame ms': after[0] - start} + if layout_mark: + result['layout ms'] = marks[layout_mark] - start if layout_mark in marks else None + trace = {} + for item in data.get('traces', []): + trace.setdefault((item['n'], item['ph']), item['t']) + for label, name in spans: + result[label] = (trace[(name, 'e')] - trace[(name, 's')] + if (name, 's') in trace and (name, 'e') in trace else None) + for label, name in events: + times = [item[3] for item in data.get('calls') or [] + if item[0] == 'event' and item[2] == name and item[3] >= start] + result[label] = min(times) - end if times else None + return result + + +def blocks(root): + manifest = root / 'blocks.json' + if manifest.exists(): + data = json.loads(manifest.read_text()) + rows = data['blocks'] + if len(rows) != len(data['states']) * data['repeats']: + raise ValueError('incomplete alternating block manifest') + return rows, data['rounds'] + rows = [] + for line in (root / 'blocks.txt').read_text().splitlines(): + fields = line.split() + if fields: + rows.append({'block': fields[0], 'state': fields[1], + 'status': 'pass' if 'status=pass' in fields else 'failed'}) + return rows, None + + +def separation(base, candidate, higher): + better = min(candidate) > max(base) if higher else max(candidate) < min(base) + worse = max(candidate) < min(base) if higher else min(candidate) > max(base) + if better: + return 'better-separated' + if worse: + return 'worse-separated' + return 'not-separated' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('root', type=Path) + parser.add_argument('baseline') + parser.add_argument('candidate') + parser.add_argument('--scenario', action='append', default=[], metavar='LABEL=RELATIVE_GLOB') + parser.add_argument('--rounds', type=int, help='required for legacy blocks.txt exports') + parser.add_argument('--start-mark', default='press') + parser.add_argument('--end-mark', default='open-complete', help='comma-separated required marks; latest first occurrence wins') + parser.add_argument('--layout-mark') + parser.add_argument('--span', action='append', default=[], metavar='LABEL=SPAN_NAME') + parser.add_argument('--event', action='append', default=[], metavar='LABEL=EVENT_KEY') + parser.add_argument('--json', type=Path) + args = parser.parse_args() + scenarios = [entry.split('=', 1) for entry in args.scenario or ['open=data/timed-*.json']] + spans = [entry.split('=', 1) for entry in args.span] + events = [entry.split('=', 1) for entry in args.event] + if any(len(pair) != 2 or not all(pair) for pair in scenarios + spans + events): + parser.error('scenario/span/event options require LABEL=VALUE') + rows, recorded_rounds = blocks(args.root) + rounds = args.rounds or recorded_rounds + if not rounds or rounds < 1 or (recorded_rounds and rounds != recorded_rounds): + parser.error('supply the planned --rounds; it must match blocks.json') + if not rows or any(row['status'] != 'pass' for row in rows): + raise ValueError('every planned block must pass; retain failed captures separately') + states = [args.baseline, args.candidate] + if set(row['state'] for row in rows) != set(states): + raise ValueError('block states do not match the requested comparison') + if any(row['state'] != states[index % 2] for index, row in enumerate(rows)): + raise ValueError('blocks must alternate baseline/candidate') + if any(sum(row['state'] == state for row in rows) < 2 for state in states): + raise ValueError('at least two blocks per state are required for the noise check') + output = {} + for scenario, pattern in scenarios: + per_state = {state: [] for state in states} + for row in rows: + paths = sorted((args.root / row['block']).glob(pattern)) + if len(paths) != rounds: + raise ValueError(f'{row["block"]}/{scenario}: expected {rounds} opens, found {len(paths)}') + samples = [timings(p, args.start_mark, args.end_mark, args.layout_mark, spans, events) + for p in paths] + per_state[row['state']].append({'block': row['block'], 'samples': samples}) + metrics = {} + for key in per_state[states[0]][0]['samples'][0]: + summaries = {} + for state in states: + values = [sample[key] for block in per_state[state] for sample in block['samples'] + if sample[key] is not None] + medians = [statistics.median(values) if values else None for values in + [[sample[key] for sample in block['samples'] if sample[key] is not None] + for block in per_state[state]]] + summaries[state] = {'n': len(values), 'p50': statistics.median(values) if values else None, + 'p75': percentile(values, .75) if values else None, + 'blockP50': medians} + complete = all(summary['n'] == rounds * len(per_state[state]) + for state, summary in summaries.items()) + noise = (separation(summaries[states[0]]['blockP50'], summaries[states[1]]['blockP50'], key == 'fps') + if complete and key not in dict(events) else 'not-checked') + metrics[key] = {'states': summaries, 'blockSeparation': noise} + output[scenario] = {'blocks': per_state, 'metrics': metrics} + print(f'\n### {scenario}\n\n| metric (p50 / p75) | {states[0]} | {states[1]} | block check |') + print('| --- | --- | --- | --- |') + for key, metric in metrics.items(): + cells = [] + for state in states: + summary = metric['states'][state] + fmt = (lambda n: f'{n:.1f}') if key == 'fps' else (lambda n: f'{n:,.0f}') + cells.append(f'{fmt(summary["p50"])} / {fmt(summary["p75"])} (n={summary["n"]})' + if summary['n'] else 'missing (n=0)') + print(f'| {key} | {cells[0]} | {cells[1]} | {metric["blockSeparation"]} |') + if args.json: + args.json.write_text(json.dumps(output, indent=2) + '\n') + + +if __name__ == '__main__': + try: + main() + except (ValueError, KeyError) as error: + print(str(error), file=sys.stderr) + sys.exit(1) diff --git a/domains/agentic/skills/recipe-performance/scripts/tests/capture-extension-window.test.sh b/domains/agentic/skills/recipe-performance/scripts/tests/capture-extension-window.test.sh new file mode 100755 index 00000000..a91a6261 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/tests/capture-extension-window.test.sh @@ -0,0 +1,142 @@ +#!/usr/bin/env bash +set -euo pipefail + +skill="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +script="$skill/scripts/capture-extension-window.mjs" + +node --check "$script" +"$script" --help 2>&1 | grep -q '^Usage: capture-extension-window\.mjs \[--cdp-port \]' +SCRIPT="$script" node --input-type=module <<'NODE' +const { normalizeUrl, sanitize } = await import(`file://${process.env.SCRIPT}`); +const canaries = [ + 'url-user-canary', 'url-pass-canary', 'query-canary', 'fragment-canary', + '0x1234567890abcdef1234567890abcdef', 'bearer-canary', 'password-canary', + 'token-canary', 'access-token-canary', 'api-key-canary', 'authorization-canary', + 'cookie-canary', 'session-canary', 'credential-canary', +]; +const url = normalizeUrl(`https://url-user-canary:url-pass-canary@example.com/accounts/0x1234567890abcdef1234567890abcdef?token=query-canary#fragment-canary`); +const evidence = JSON.stringify(sanitize({ + error: `request failed at ${url}; Authorization: Bearer bearer-canary; password=password-canary`, + requestUrl: `https://example.com/session/0x1234567890abcdef1234567890abcdef?key=query-canary#fragment-canary`, + token: 'token-canary', + accessToken: 'access-token-canary', + apiKey: 'api-key-canary', + authorization: 'authorization-canary', + cookie: 'cookie-canary', + session: { id: 'session-canary' }, + credential: 'credential-canary', +})); +for (const canary of canaries) { + if (url.includes(canary) || evidence.includes(canary)) throw new Error(`secret canary leaked: ${canary}`); +} +if (!url.includes('example.com') || !evidence.includes('[REDACTED')) throw new Error('sanitized evidence lost safe context or its redaction marker'); +NODE +SCRIPT="$script" node --input-type=module <<'NODE' +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { spawnSync } from 'node:child_process'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +const { run, terminateWithSignal } = await import(`file://${process.env.SCRIPT}`); +const root = mkdtempSync(path.join(tmpdir(), 'capture-extension-window-')); +const commandWithDescendant = (pidPath, resistSigterm = false) => [ + process.execPath, + '-e', + `const fs = require('node:fs'); + const { spawn } = require('node:child_process'); + const child = spawn(process.execPath, ['-e', ${JSON.stringify( + resistSigterm ? "process.on('SIGTERM', () => {}); setInterval(() => {}, 1000)" : 'setInterval(() => {}, 1000)', + )}], { stdio: 'ignore' }); + fs.writeFileSync(process.argv[1], String(child.pid)); + setInterval(() => {}, 1000);`, + pidPath, +]; +const assertStopped = async (pidPath) => { + const descendantPid = Number(readFileSync(pidPath, 'utf8')); + try { + process.kill(descendantPid, 0); + throw new Error('measured command left a descendant running'); + } catch (error) { + if (error?.code !== 'ESRCH') throw error; + } +}; +try { + const timeoutPidPath = path.join(root, 'timeout-descendant.pid'); + const startedAt = Date.now(); + try { + await run(commandWithDescendant(timeoutPidPath), 500); + throw new Error('hanging measured command unexpectedly passed'); + } catch (error) { + if (!error.message.includes('measured command timed out after 500ms')) throw error; + if (Date.now() - startedAt > 2_000) throw new Error('timed-out measured command was not terminated'); + await assertStopped(timeoutPidPath); + } + + for (const signal of ['SIGHUP', 'SIGTERM']) { + const signalPidPath = path.join(root, `${signal}-descendant.pid`); + try { + setTimeout(() => process.kill(process.pid, signal), 500); + await run(commandWithDescendant(signalPidPath, signal === 'SIGTERM'), 5_000); + throw new Error('interrupted measured command unexpectedly passed'); + } catch (error) { + if (!error.message.includes(`measured command interrupted by ${signal}`)) throw error; + await assertStopped(signalPidPath); + } + } +} finally { + rmSync(root, { recursive: true, force: true }); +} + +const statusProbe = spawnSync(process.execPath, [ + '--input-type=module', + '-e', + `const { run, terminateWithSignal } = await import(${JSON.stringify(`file://${process.env.SCRIPT}`)}); + setTimeout(() => process.kill(process.pid, 'SIGTERM'), 100); + try { + await run([process.execPath, '-e', 'setInterval(() => {}, 1000)'], 5000); + } catch (error) { + terminateWithSignal(error.signal); + }`, +]); +if (statusProbe.signal !== 'SIGTERM') { + throw new Error(`interrupted wrapper lost SIGTERM status: ${statusProbe.status}/${statusProbe.signal}`); +} +if (typeof terminateWithSignal !== 'function') throw new Error('signal termination helper is unavailable'); +NODE +! grep -q 'command: options.command' "$script" + +set +e +output="$("$script" --cdp-port 9500 --boundary body --out-dir /tmp 2>&1)" +status=$? +set -e +test "$status" -eq 2 +grep -q '^capture-extension-window: pass the measured command after --$' <<<"$output" + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT +mkdir -p "$tmp/temp/recipe/runtime" +printf '{"cdpPort":1}\n' > "$tmp/temp/recipe/runtime/agentic-runtime.json" +set +e +output="$(cd "$tmp" && "$script" --boundary body --out-dir artifacts -- true 2>&1)" +status=$? +set -e +test "$status" -eq 1 +grep -q 'fetch failed' <<<"$output" + +printf '{"cdpPort":0}\n' > "$tmp/temp/recipe/runtime/agentic-runtime.json" +set +e +output="$(cd "$tmp" && "$script" --boundary body --out-dir artifacts -- true 2>&1)" +status=$? +set -e +test "$status" -eq 1 +grep -q 'no authoritative Extension CDP port found' <<<"$output" + +printf '{broken json\n' > "$tmp/temp/recipe/runtime/agentic-runtime.json" +set +e +output="$(cd "$tmp" && "$script" --cdp-port 1 --boundary body --out-dir artifacts -- true 2>&1)" +status=$? +set -e +test "$status" -eq 1 +grep -q 'fetch failed' <<<"$output" +! grep -q 'could not read .*agentic-runtime.json' <<<"$output" + +echo 'capture-extension-window: CLI contract passed' diff --git a/domains/agentic/skills/recipe-performance/scripts/tests/paired-device.test.mjs b/domains/agentic/skills/recipe-performance/scripts/tests/paired-device.test.mjs new file mode 100644 index 00000000..37a19105 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/tests/paired-device.test.mjs @@ -0,0 +1,125 @@ +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { fileURLToPath } from 'node:url'; + +const scripts = fileURLToPath(new URL('..', import.meta.url)); +function fixture(t) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mobile-perf-')); + t.after(() => fs.rmSync(dir, { recursive: true })); + return dir; +} +function python(script, args) { + return spawnSync('python3', [path.join(scripts, script), ...args], { encoding: 'utf8' }); +} + +test('blocks alternate and a failed capture stops without overwriting its evidence', (t) => { + const dir = fixture(t); + const adapter = path.join(dir, 'capture'); + fs.writeFileSync(adapter, '#!/usr/bin/env python3\nimport sys\nfrom pathlib import Path\np=Path(sys.argv[3]); (p/"observed.txt").write_text(" ".join(sys.argv[1:]))\nsys.exit(7 if sys.argv[1]=="broken" else 0)\n', { mode: 0o755 }); + const out = path.join(dir, 'ok'); + const result = python('paired-blocks.py', [out, '5', '2', 'base:abc', 'head:def', adapter]); + assert.equal(result.status, 0, result.stderr); + const manifest = JSON.parse(fs.readFileSync(path.join(out, 'blocks.json'))); + assert.deepEqual(manifest.blocks.map(x => [x.block, x.state, x.requestedRef]), + [['a1', 'base', 'abc'], ['b1', 'head', 'def'], ['a2', 'base', 'abc'], ['b2', 'head', 'def']]); + assert.equal(python('paired-blocks.py', [out, '5', '2', 'base:abc', 'head:def', adapter]).status, 1); + const failed = path.join(dir, 'failed'); + assert.equal(python('paired-blocks.py', [failed, '5', '2', 'base:abc', 'broken:def', adapter]).status, 7); + const partial = JSON.parse(fs.readFileSync(path.join(failed, 'blocks.json'))); + assert.deepEqual(partial.blocks.map(x => x.status), ['pass', 'failed']); + assert.ok(fs.existsSync(path.join(failed, 'b1/observed.txt'))); +}); + +function captures(dir, baseGaps, headGaps) { + const blocks = []; + for (const [index, state, gap] of [[1, 'base', baseGaps[0]], [1, 'head', headGaps[0]], + [2, 'base', baseGaps[1]], [2, 'head', headGaps[1]]]) { + const block = `${state === 'base' ? 'a' : 'b'}${index}`; + const data = path.join(dir, block, 'data'); fs.mkdirSync(data, { recursive: true }); + for (const [n, value] of gap.entries()) { + const raw = { marks: [{ n: 'press', t: 1000 }, { n: 'open-complete', t: 2000 }], + frames: [900, 1000 + value, 2050] }; + fs.writeFileSync(path.join(data, `timed-${n}.json`), JSON.stringify(JSON.stringify(raw))); + } + blocks.push({ block, state, status: 'pass' }); + } + fs.writeFileSync(path.join(dir, 'blocks.json'), JSON.stringify({ rounds: 2, repeats: 2, + states: [['base', 'abc'], ['head', 'def']], blocks })); +} + +test('interpolated percentiles retain a pooled gain that is not separated by blocks', (t) => { + const dir = fixture(t); + captures(dir, [[100, 200], [110, 210]], [[50, 100], [120, 220]]); + const output = path.join(dir, 'metrics.json'); + const result = python('report.py', [dir, 'base', 'head', '--json', output]); + assert.equal(result.status, 0, result.stderr); + const metric = JSON.parse(fs.readFileSync(output)).open.metrics['next JS frame ms']; + assert.equal(metric.states.base.p50, 155); + assert.equal(metric.states.base.p75, 202.5); + assert.equal(metric.states.head.p50, 110); + assert.equal(metric.blockSeparation, 'not-separated'); + const file = path.join(dir, 'a1/data/timed-0.json'); + const raw = JSON.parse(JSON.parse(fs.readFileSync(file))); + raw.marks.pop(); fs.writeFileSync(file, JSON.stringify(raw)); + const invalid = python('report.py', [dir, 'base', 'head']); + assert.notEqual(invalid.status, 0); + assert.match(invalid.stderr, /missing or invalid press\/open-complete/); +}); + +test('whole overlapping frame gaps and a higher-FPS direction are preserved', (t) => { + const dir = fixture(t); + captures(dir, [[100, 100], [100, 100]], [[50, 50], [50, 50]]); + const raw = { marks: [{ n: 'press', t: 1000 }, { n: 'open-complete', t: 2000 }], + frames: [900, 1100, 1150, 2200] }; + for (const block of ['a1', 'b1', 'a2', 'b2']) { + for (const n of [0, 1]) { + const sample = block.startsWith('b') ? { ...raw, frames: [900, 1100, 1150, 1500, 2200] } : raw; + fs.writeFileSync(path.join(dir, block, `data/timed-${n}.json`), JSON.stringify(sample)); + } + } + const output = path.join(dir, 'metrics.json'); + assert.equal(python('report.py', [dir, 'base', 'head', '--json', output]).status, 0); + const metrics = JSON.parse(fs.readFileSync(output)).open.metrics; + assert.equal(metrics.fps.states.base.p50, 2); + assert.equal(metrics.fps.states.head.p50, 3); + assert.equal(metrics.fps.blockSeparation, 'better-separated'); + assert.equal(metrics['>100 ms frames'].states.base.p50, 2); + assert.equal(metrics['longest frame'].states.base.p50, 1050); + assert.equal(metrics['blocked ms'].states.base.p50, 1150); + assert.equal(metrics['next JS frame ms'].states.base.p50, 100); +}); + +test('completion waits for both slide and first content layout, excluding later relayouts', (t) => { + const dir = fixture(t); + captures(dir, [[100, 100], [100, 100]], [[50, 50], [50, 50]]); + for (const block of ['a1', 'b1', 'a2', 'b2']) { + for (const n of [0, 1]) { + const file = path.join(dir, block, `data/timed-${n}.json`); + const raw = JSON.parse(JSON.parse(fs.readFileSync(file))); + raw.marks.push({ n: 'content-layout', t: 2100 }, { n: 'content-layout', t: 3500 }); + raw.frames.push(2250); + fs.writeFileSync(file, JSON.stringify(raw)); + } + } + const output = path.join(dir, 'metrics.json'); + const result = python('report.py', [dir, 'base', 'head', '--end-mark', 'open-complete,content-layout', '--json', output]); + assert.equal(result.status, 0, result.stderr); + assert.equal(JSON.parse(fs.readFileSync(output)).open.metrics['window ms'].states.base.p50, 1100); +}); + +test('captures must bracket both boundaries so a collection cutoff cannot hide a long gap', (t) => { + const dir = fixture(t); + captures(dir, [[100, 100], [100, 100]], [[50, 50], [50, 50]]); + const file = path.join(dir, 'a1/data/timed-0.json'); + const raw = JSON.parse(JSON.parse(fs.readFileSync(file))); + for (const frames of [[1100, 1500, 2200], [900, 1100, 1500, 1900]]) { + fs.writeFileSync(file, JSON.stringify({ ...raw, frames })); + const result = python('report.py', [dir, 'base', 'head']); + assert.notEqual(result.status, 0); + assert.match(result.stderr, /frame probe did not cover the open/); + } +}); diff --git a/domains/agentic/skills/recipe-performance/scripts/tests/profile-flame.test.mjs b/domains/agentic/skills/recipe-performance/scripts/tests/profile-flame.test.mjs new file mode 100644 index 00000000..e5b1edd8 --- /dev/null +++ b/domains/agentic/skills/recipe-performance/scripts/tests/profile-flame.test.mjs @@ -0,0 +1,60 @@ +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import test from 'node:test'; +import { fileURLToPath } from 'node:url'; + +const script = fileURLToPath(new URL('../profile-flame.py', import.meta.url)); +test('CPU view weights the caller tree, escapes labels and rejects empty profiles', t => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'profile-flame-')); + t.after(() => fs.rmSync(dir, { recursive: true })); + const input = path.join(dir, 'sample.cpuprofile'); + const output = path.join(dir, 'flame.html'); + const profile = { + nodes: [ + { id: 1, callFrame: { functionName: '(root)' }, children: [2, 3] }, + { id: 2, callFrame: { functionName: '', url: 'http://local/a.js?token=private', lineNumber: 9 } }, + { id: 3, callFrame: { functionName: 'layout' } }, + ], + samples: [2, 2, 3], timeDeltas: [1000, 2000, 1000], startTime: 0, endTime: 4000, + }; + fs.writeFileSync(input, JSON.stringify(profile)); + const run = () => spawnSync('python3', [script, input, output], { encoding: 'utf8' }); + assert.equal(run().status, 0); + const html = fs.readFileSync(output, 'utf8'); + assert.match(html, /width="900.000"/); + assert.match(html, /<mount> a.js:10: inclusive 3.000 ms; self 3.000 ms/); + assert.doesNotMatch(html, /token|private|/); + assert.notEqual(run().status, 0, 'must not overwrite evidence'); + fs.unlinkSync(output); + fs.writeFileSync(input, JSON.stringify({ ...profile, samples: [], timeDeltas: [] })); + const empty = run(); + assert.notEqual(empty.status, 0); + assert.match(empty.stderr, /CPU samples and matching timeDeltas are required/); + assert.equal(fs.existsSync(output), false); +}); + + +test('deep sampled stacks render without changing the interpreter recursion limit', t => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'profile-flame-deep-')); + t.after(() => fs.rmSync(dir, { recursive: true })); + const count = 1500; + const profile = { + nodes: Array.from({ length: count }, (_, id) => ({ + id, callFrame: { functionName: `frame${id}` }, + children: id + 1 < count ? [id + 1] : [], + })), + samples: [count - 1], timeDeltas: [1000], startTime: 0, endTime: 1000, + }; + const input = path.join(dir, 'deep.cpuprofile'); + const output = path.join(dir, 'deep.html'); + fs.writeFileSync(input, JSON.stringify(profile)); + const result = spawnSync('python3', [script, input, output], { encoding: 'utf8' }); + assert.equal(result.status, 0, result.stderr); + const html = fs.readFileSync(output, 'utf8'); + assert.equal((html.match(/ Using the recipe-performance skill, make the Perps order sheet (tap Long or Short on a market) open faster. Find the biggest bottleneck, fix it, and give before/after stats from the device. + +Expect a bottleneck diagnosis backed by a CPU profile and viewable flame graph, +a focused code change when requested, and paired before/after evidence with n, +p50/p75, frame gaps and a noise verdict. The report records the target, source, +build/sampler configuration, raw artifacts and actual wall time. + +For planning, budget about 5–15 minutes for a prepared four-block comparison per +flow. First-time fixture setup, instrumentation and bisection can take hours; +that capture budget is not an end-to-end time guarantee. + +## Interactive checkpoints + +Let Cook initialize and prepare the task before product discovery. Then present the proposed user journey, exact measured start/end boundary, client/lifecycle/environment, evidence authorities, failure verdict, and whether the run is baseline-only or may lead to a fix. A request to measure, diagnose, or compare a named flow authorizes its baseline/comparison measurement. Confirm only an unresolved scope or environment choice before measuring. + +Choose the path the request needs. A diagnosis asks for bottleneck evidence and a proposed fix; an existing-PR or null comparison goes directly to paired measurement, without requiring a new fix. + +After the baseline, present the verdict and evidence. Diagnosis may use reversible task-local timing probes and elimination variants, with their patches retained and restored. If the caller requested a fix or optimization, apply the smallest change supported by that baseline and rerun the identical recipe. For a diagnosis-only request, present the proposed fix for approval first. + +## Contract first + +Write one row per claim before editing: + +| Claim | Real user action | Exact start/end boundary | Context/tags | Authority | Failure verdict | +|---|---|---|---|---|---| + +Resolve names, routes, selectors, controller methods, and telemetry attributes from current source or live CLI discovery. Never reuse a historical name without confirming it exists. + +## Workflow + +1. Apply the task harness's `help --json` method to select, compose, or repair the smallest recipe for the claim. Keep reusable domain composition in a team library; do not grow `mm-harness` for one investigation. +2. Make preparation deterministic but outside the measured window. Use real lifecycle actions for cold, warm, and background-resume states. Use controller actions for state, assertions, and cleanup—not as substitutes for a claimed UI gesture. +3. Run one baseline recipe. Runtime claims require the real app and a run-scoped assertion. Focused tests corroborate; they never replace runtime proof. +4. Match evidence to the claim: + - recipe `trace.json`: graph execution and orchestration duration; + - exact product start/end markers: local CUF boundary and tags; + - Sentry spans/dashboard query: indexed telemetry and aggregate latency; + - Chrome DevTools/CDP trace: Extension JavaScript/render/network diagnosis; + - Mobile JS CPU sampling profile plus rendered flame graph: function attribution around the recipe action; native frame evidence distinguishes UI rendering from JS stalls; + - controller read: resulting application state. +5. For a Mobile diagnosis, correlate the CPU profile with JS frame gaps and source, then use blank-then-add elimination or mount-read runtime switches to test the cause. Follow the Mobile reference. +6. On an unexpected result, stop and classify `PROVEN`, `DISPROVEN`, `INCONCLUSIVE`, or `BLOCKED`. Do not broaden queries, change actions, or retry a nondeterministic path to manufacture a pass. +7. Fix only a proven defect within the authorised scope, then rerun the identical recipe and evidence query. Cleanup must leave the selected account/market deterministic even after failure. + +## Efficiency gate + +Add a reusable recipe or instruction only when it removes repeated inference, encodes a safety/evidence invariant, or has already served multiple proofs. Prefer direct, discoverable controller/CDP use for one-off diagnosis. Never add an action merely to hide a standard tool the agent can already use reliably. + +## Output + +Retain the contract matrix, root recipe, resolved dependency recipes and resolution graph, trace, diagnostics, exact marker excerpts, controller state, and redacted Sentry/trace summary. For Mobile, retain the raw CPU profile, viewable flame graph, elimination conditions and paired block table where applicable. Separate observations from inference; do not claim cross-platform parity unless equivalent boundaries are verified on each client.