feat: add automated tool-usage collection via Yarn plugin and agent hooks (Phase 2)

[NicolasMassart]

Description

Phase 2 of the automated tool-usage collection infrastructure (MCWP-513), building on the core DB write infrastructure from Phase 1.

What changed:

• Yarn Berry plugin (plugin-usage-tracking.cjs): automatically records start/end/interrupted events for every yarn <script> execution via the wrapScriptExecution hook — no changes to package.json script entries required. Fires-and-forgets via a detached tsx subprocess so it never blocks the terminal. Exit code 129 (SIGHUP/Ctrl+C) is recorded as interrupted with success=NULL to distinguish abandoned sessions from genuine failures.
• Claude Code PreToolUse hook (.claude/skills/pr-changelog/SKILL.md): frontmatter hook fires automatically on the first tool call after the skill loads — zero tokens, agent-invisible.
• Cursor beforeReadFile hook (.cursor/hooks.json + cursor-hook-skill-tracking.ts): fires automatically when Cursor reads any .agents/skills/*/SKILL.md — zero tokens, agent-invisible. Uses execFileSync with an explicit argument array to avoid shell injection. Guards file_path type before extracting the skill name.
• CI guard + developer opt-out (TOOL_USAGE_COLLECTION_OPT_IN=false): all three collection paths (Yarn plugin, Claude hook, Cursor hook) skip silently when CI is set or when TOOL_USAGE_COLLECTION_OPT_IN=false is exported in the developer's shell profile. The Yarn plugin returns an empty factory; the shell commands use [ -z "$CI" ] && [ "$TOOL_USAGE_COLLECTION_OPT_IN" != "false" ] && guards. The Cursor hook uses || echo '{"permission":"allow"}' as a fallback so Cursor always receives an explicit JSON permission response, even when the guards short-circuit.
• Developer documentation (scripts/tooling/README.md, AGENTS.md, docs/readme/development-process.md): documents all three collection paths, skip conditions, and the architecture with a Mermaid diagram; clarifies that data is recorded locally only, never sent outside.

Changelog

CHANGELOG entry: null

Related issues

Fixes MCWP-513

Manual testing steps

Prerequisites: branch checked out, yarn install done, sqlite3 available (macOS: pre-installed; Linux: sudo apt install sqlite3 or brew install sqlite3), Claude Code installed and configured with this repo as the working directory (required for Path 2 only — https://docs.anthropic.com/en/docs/claude-code)

Feature: automated tool-usage collection (Phase 2)

  Background:
    Given the branch is checked out and dependencies are installed with `yarn install`
    And sqlite3 is available in the terminal
      # macOS: pre-installed. Linux: sudo apt install sqlite3 or brew install sqlite3
    And Claude Code is installed and configured with this repo as the working directory
      # required for Path 2 only — https://docs.anthropic.com/en/docs/claude-code
    And the Cursor `beforeReadFile` hook is active
      # required for Path 3 only — hooks.json is committed; Cursor loads it automatically for trusted workspaces

  # ── Path 1: Yarn Berry plugin ──────────────────────────────────────────────

  Scenario: Yarn plugin records start and end events for a successful script run
    Given no recent events exist in ~/.tool-usage-collection/events.db for tool_name "yarn:setup:expo"
      # check: sqlite3 ~/.tool-usage-collection/events.db "SELECT * FROM events WHERE tool_name='yarn:setup:expo' ORDER BY created_at DESC LIMIT 3;"
      # clean: sqlite3 ~/.tool-usage-collection/events.db "DELETE FROM events WHERE tool_name='yarn:setup:expo';"
    When user runs `yarn setup:expo`
    Then 2 new rows appear in the events table with tool_name = "yarn:setup:expo" and tool_type = "yarn_script"
      # verify: sqlite3 ~/.tool-usage-collection/events.db "SELECT event_type, tool_type, success, duration_ms FROM events WHERE tool_name='yarn:setup:expo' ORDER BY created_at DESC LIMIT 2;"
      # expected output:
      #   end|yarn_script|1|<ms>
      #   start|yarn_script||
    And one row has event_type = "start"
    And one row has event_type = "end" with success = 1 and duration_ms > 0

  Scenario: Yarn plugin records an interrupted event (not end) when a script is cancelled with Ctrl+C
    Given no recent events exist for tool_name "yarn:test:unit"
      # check: sqlite3 ~/.tool-usage-collection/events.db "SELECT * FROM events WHERE tool_name='yarn:test:unit' ORDER BY created_at DESC LIMIT 3;"
      # clean: sqlite3 ~/.tool-usage-collection/events.db "DELETE FROM events WHERE tool_name='yarn:test:unit';"
    When user runs `yarn test:unit` and interrupts it with Ctrl+C before completion
      # Note: Yarn terminates the child via SIGHUP (exit code 129) on Ctrl+C — the plugin
      # detects this and writes event_type='interrupted' instead of 'end', with success=NULL
    Then 2 rows exist for tool_name = "yarn:test:unit"
      # verify: sqlite3 ~/.tool-usage-collection/events.db "SELECT event_type, success, duration_ms FROM events WHERE tool_name='yarn:test:unit' ORDER BY created_at DESC LIMIT 2;"
      # expected output (duration_ms varies, success is always empty for interrupted):
      #   interrupted||<ms>
      #   start||
    And one row has event_type = "start" and one row has event_type = "interrupted"
    And the "interrupted" row has success = NULL and duration_ms > 0

  Scenario: Yarn plugin skips collection when CI env var is set
    Given CI=true is set in the environment
    When user runs any `yarn` script
    Then no new rows are written to ~/.tool-usage-collection/events.db
      # The plugin returns an empty factory immediately — no hooks registered, no subprocess spawned

  Scenario: Yarn plugin skips collection when opted out
    Given TOOL_USAGE_COLLECTION_OPT_IN=false is exported in the shell
    When user runs any `yarn` script
    Then no new rows are written to ~/.tool-usage-collection/events.db

  # ── Path 2: Claude Code skill (PreToolUse hook) ──────────────────────────

  Scenario: Claude Code PreToolUse hook fires automatically when the pr-changelog skill is invoked
    Given no recent events exist for tool_name "skill:pr-changelog" and agent_vendor "claude"
      # check: sqlite3 ~/.tool-usage-collection/events.db "SELECT * FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='claude' ORDER BY created_at DESC LIMIT 3;"
      # clean: sqlite3 ~/.tool-usage-collection/events.db "DELETE FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='claude';"
    When user starts a Claude Code session in this repo and asks "write a changelog entry for this branch"
      # The PreToolUse hook in .claude/skills/pr-changelog/SKILL.md fires automatically
      # on the first tool call after the skill is loaded (once: true) — the agent does not need to call anything explicitly
    Then 1 new row exists with tool_name = "skill:pr-changelog", tool_type = "skill", event_type = "start", agent_vendor = "claude"
      # verify: sqlite3 ~/.tool-usage-collection/events.db "SELECT tool_name, tool_type, event_type, agent_vendor FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='claude' ORDER BY created_at DESC LIMIT 1;"
      # expected output:
      #   skill:pr-changelog|skill|start|claude

  # ── Path 3: Cursor beforeReadFile hook ────────────────────────────────────

  Scenario: Cursor beforeReadFile hook records a start event when a skill file is read
    Given no recent events exist for tool_name "skill:pr-changelog" and agent_vendor "cursor"
      # check: sqlite3 ~/.tool-usage-collection/events.db "SELECT * FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='cursor' ORDER BY created_at DESC LIMIT 3;"
      # clean: sqlite3 ~/.tool-usage-collection/events.db "DELETE FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='cursor';"
    When user opens a Cursor chat in this repo and asks "write a changelog entry for this branch"
      # Cursor reads .agents/skills/pr-changelog/SKILL.md — the beforeReadFile hook fires automatically
      # cursor-hook-skill-tracking.ts extracts "pr-changelog" from the path and calls tool-usage-collection.ts
      # The agent does not need to call anything explicitly — zero tokens
    Then 1 new row exists with tool_name = "skill:pr-changelog", tool_type = "skill", event_type = "start", agent_vendor = "cursor"
      # verify: sqlite3 ~/.tool-usage-collection/events.db "SELECT tool_name, tool_type, event_type, agent_vendor FROM events WHERE tool_name='skill:pr-changelog' AND agent_vendor='cursor' ORDER BY created_at DESC LIMIT 1;"
      # expected output:
      #   skill:pr-changelog|skill|start|cursor

Screenshots/Recordings

Before

N/A

After

N/A

Pre-merge author checklist

• I've followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
• I've completed the PR template to the best of my ability
• I've included tests if applicable
• I've documented my code using JSDoc format if applicable
• I've applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

• I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
• I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

---

Note

Medium Risk
Touches developer build tooling by hooking into every yarn script execution and spawning detached subprocesses; while guarded for CI/opt-out, misconfigurations could impact local dev workflows.

Overview
Adds automated local developer tool-usage collection across Yarn scripts and AI agent skills, writing start/end/interrupted events to ~/.tool-usage-collection/events.db with opt-out via TOOL_USAGE_COLLECTION_OPT_IN=false and automatic CI disable.

Introduces a Yarn Berry plugin that wraps yarn <script> runs and fire-and-forgets a detached tsx subprocess for tracking, plus new Cursor and Claude hooks to record skill usage; updates the SQLite schema/CLI/tests to support the new interrupted event type and expands documentation for setup/inspection.

^{Reviewed by Cursor Bugbot for commit 0877b05. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.20%. Comparing base (15470f4) to head (9e1c2c0).
⚠️ Report is 53 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #28815      +/-   ##
==========================================
- Coverage   82.20%   82.20%   -0.01%     
==========================================
  Files        5017     5023       +6     
  Lines      131659   131903     +244     
  Branches    29381    29445      +64     
==========================================
+ Hits       108228   108428     +200     
- Misses      16083    16110      +27     
- Partials     7348     7365      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Cal-L]

Lgtm

[NicolasMassart]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Resolved by merging main into this branch and fixing the conflict in scripts/tooling/README.md. Commit: f86ebe3.

[github-actions]

🔍 Smart E2E Test Selection

• Selected E2E tags: None (no tests recommended)
• Selected Performance tags: None (no tests recommended)
• Risk Level: low
• AI Confidence: 98%

click to see 🤖 AI reasoning details

E2E Test Selection:
All 16 changed files are developer tooling and documentation changes with zero impact on the MetaMask Mobile app:

1. scripts/tooling/*.ts - New TypeScript scripts for tracking AI agent tool/skill usage (Cursor hooks, event tracking, DB utilities, CLI tool). These are developer experience tools that run locally and have no connection to the app code.
2. scripts/tooling/*.test.ts - Unit tests for the above tooling scripts.
3. .cursor/hooks.json - Cursor IDE hook configuration for the skill tracking feature.
4. .yarn/plugins/plugin-usage-tracking.cjs - Yarn plugin for usage tracking.
5. .yarnrc.yml - Yarn configuration update.
6. AGENTS.md - Documentation for AI coding agents.
7. docs/readme/development-process.md - Development process documentation.
8. scripts/tooling/README.md - README for the tooling scripts.
9. .claude/skills/pr-changelog/SKILL.md - Claude skill documentation.
10. yarn.lock - Dependency lock file update (for the new yarn plugin).

None of these files touch app source code, E2E test infrastructure, CI/CD workflows, controllers, Engine, navigation, UI components, or any wallet functionality. No E2E tests are needed.

Performance Test Selection:
No performance-relevant changes. All changes are developer tooling scripts, documentation, and configuration files that have no impact on app rendering, data loading, state management, or any user-facing functionality.

View GitHub Actions results

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Cal-L]

Lgtm