Skip to content

Commit 06a417e

Browse files
author
Nikhil Agrawal
committed
feat(mobile-apps): emit skill-started telemetry from plugin hooks
Wires the Mobile Apps plugin into the vendored 1DS telemetry library so we can see which mobile skills developers actually run. - Register two hooks: PreToolUse(Skill) for agent-invoked skills and UserPromptSubmit for manual slash commands, which Copilot pre-expands into <skill-context> and never reports as a Skill tool call. - Add mobile-telemetry.js, the plugin adapter that gates on a provisioned ikey, resolves the session, and builds the skill-started event. - Correlate nested Copilot agents, which receive a transient call_* session id, back to their owning UUID session via a bounded tail read of the local session-state log, cached as a short-lived alias file. Fails open. - Discover tracked skills from the skills directory instead of hardcoding. - Add the telemetry skill so users can inspect or opt out. - Add script tests plus a CI workflow that sets the plugin opt-out env var so test runs never post to the production collector.
1 parent 9aedd3c commit 06a417e

12 files changed

Lines changed: 1235 additions & 4 deletions

File tree

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Mobile Apps telemetry is exercised with isolated configs and local mirrors.
2+
# The job-level opt-out is a backstop against accidental production emission if
3+
# the plugin is provisioned later and a new test forgets its local test seam.
4+
name: mobile-apps-script-tests
5+
6+
on:
7+
pull_request:
8+
branches:
9+
- main
10+
paths:
11+
- "plugins/mobile-apps/**"
12+
- "shared/telemetry/**"
13+
14+
jobs:
15+
test-mobile-apps-scripts:
16+
name: test-mobile-apps-scripts (${{ matrix.os }})
17+
runs-on: ${{ matrix.os }}
18+
env:
19+
POWER_PLATFORM_SKILLS_TELEMETRY_MOBILE_APP_OPTOUT: "1"
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
os:
24+
- ubuntu-latest
25+
- windows-latest
26+
- macos-latest
27+
steps:
28+
- name: checkout
29+
uses: actions/checkout@v4
30+
31+
- name: setup-node
32+
uses: actions/setup-node@v4
33+
with:
34+
node-version: 22
35+
36+
- name: run-mobile-apps-script-tests
37+
shell: bash
38+
working-directory: plugins/mobile-apps/scripts/tests
39+
run: node --test

plugins/mobile-apps/AGENTS.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
This file provides guidance to AI Agents when working with the **mobile-app** plugin.
44

5-
> **Status:** v0 — 23 skills + 5 agents authored. The latest Expo standalone template snapshot is bundled under `template/`. Read [README.md](./README.md) for the command list.
5+
> **Status:** v0 — 24 skills + 5 agents authored. The latest Expo standalone template snapshot is bundled under `template/`. Read [README.md](./README.md) for the command list.
66
77
## What This Plugin Is
88

@@ -26,8 +26,8 @@ README.md ← Plugin overview
2626
agents/ ← native-app-planner, data-model-architect, screen-planner, screen-builder
2727
shared/ ← shared-instructions, references, samples, memory-bank template
2828
skills/ ← /create-mobile-app, /add-dataverse, /add-connector, /add-native, ...
29-
scripts/ ← shared helpers, including validate-mobile-files.js for skill-owned changed-file validation
30-
hooks/ ← Validator implementations invoked explicitly by mobile workflows
29+
scripts/ ← shared helpers, including validate-mobile-files.js and bundled telemetry
30+
hooks/ ← Telemetry start hooks plus validators invoked explicitly by mobile workflows
3131
```
3232

3333
## Template source
@@ -56,7 +56,7 @@ Do not add preparation rewrites for `scheme`, `package`, `bundleIdentifier`, `sr
5656
7. **Persisted plan** — Write `native-app-plan.md` (Mermaid ER + per-screen specs + native capabilities matrix) as the source of truth that sub-skills `Read`.
5757
8. **CLI compatibility** — Use `npx power-apps ...` for code-app lifecycle and data-source commands. Use `scripts/resolve-environment.js` plus `az` tokens for Dataverse environment URL/tenant discovery and Azure/Entra operations. See [`shared/shared-instructions.md`](./shared/shared-instructions.md).
5858
9. **Agent invocation namespace** — All `Task` invocations of agents in this plugin MUST use the fully-qualified `mobile-app:<agent-name>` form (e.g. `mobile-app:native-app-planner`, `mobile-app:screen-builder`). Bare names like `native-app-planner` return `Agent type 'native-app-planner' not found` because Claude Code namespaces all plugin agents by plugin name.
59-
10. **Plugin isolation**Do not add `hooks/hooks.json`: Claude loads plugin hooks during unrelated workflows, so a mobile write hook can block Canvas Apps tool calls. Mutating skills follow the changed-file gate in `shared/shared-instructions.md`, and final-artifact agents invoke `scripts/validate-mobile-files.js` directly.
59+
10. **Plugin isolation**`hooks/hooks.json` is limited to fail-open telemetry start hooks. They never validate, mutate, or block tool calls. Do not add write/validation hooks: mutating skills follow the changed-file gate in `shared/shared-instructions.md`, and final-artifact agents invoke `scripts/validate-mobile-files.js` directly.
6060
11. **Invocation metadata** — Public entry skills use `user-invocable: true` and remain model-invocable. Bundled implementation helpers use both `user-invocable: false` and `disable-model-invocation: true`; their owner reads `SKILL.md` directly. Hidden standalone workflows such as `assign-offline-profile` and `preview-offline-scope` use `user-invocable: false` without disabling model invocation because no owner reads them directly. Agents use `user-invocable: false` without `disable-model-invocation` so qualified `Task` delegation remains available.
6161
12. **Sub-agent return-status protocol** — Every agent in this plugin (`native-app-planner`, `data-model-architect`, `screen-planner`, `screen-builder`) MUST return a status code as the **literal first line** of its final message. Orchestrators (skills that invoke agents via `Task`) MUST parse the first line and branch:
6262

@@ -74,6 +74,17 @@ Do not add preparation rewrites for `scheme`, `package`, `bundleIdentifier`, `sr
7474
- Special early-return signals (`INDUSTRY_CONFIRM_REQUESTED:`, `DESIGN_VIBE_REQUESTED:`) pre-date this protocol and remain in effect — they are special-cased "ask the user one question and re-spawn me" handoffs, not terminal returns.
7575
- The canonical orchestrator handler lives in [`skills/create-mobile-app/SKILL.md`](./skills/create-mobile-app/SKILL.md) Step 3.0. Future skills that spawn agents should reference it rather than duplicating the switch.
7676

77+
## Telemetry
78+
79+
Mobile Apps bundles the canonical stdlib-only 1DS transport from the repo-root `shared/telemetry/lib` at `scripts/lib/telemetry/lib`. Edit the shared source first, then refresh this physical copy in the same change; never copy another plugin's `ikey.json` or resolver.
80+
81+
- **Start-only lifecycle:** `UserPromptSubmit` records explicit slash-command starts and `PreToolUse(Skill)` records programmatic Skill-tool starts; both may fire for one visible slash command. `UserPromptSubmit` payloads differ by host — Claude Code passes the raw `/mobile-app:<skill>` text, Copilot CLI pre-expands it to a `<skill-context name="<skill>">` wrapper and emits no Skill pre-tool event — so both shapes must stay recognized or manual runs go uncaptured. Do not add `skill_completed`, duration, outcome, or persisted correlation state: Power Pages deliberately removed that flow because the hook boundary does not prove the workflow completed.
82+
- **Coverage and attribution:** `scripts/lib/mobileapp-hook-utils.js` discovers every user- or model-invocable top-level skill, including `telemetry`. Direct-read helpers with `disable-model-invocation: true` are not independently invoked and are excluded. Bare and `mobile-app:`-qualified names are both attributed; explicitly foreign plugin namespaces are excluded.
83+
- **Session correlation:** Stable host session ids pass through unchanged. Copilot CLI reports a transient `call_*` id to nested-agent hooks, so `resolveCopilotRootSessionId` in `scripts/lib/mobile-telemetry.js` resolves it to the unique recent UUID session whose local `~/.copilot/session-state/<uuid>/events.jsonl` structurally owns that `agentId`, reading only a bounded tail. Keep host-specific quirks contained in that one function. The verified root is cached as one atomic 30-minute alias file per hashed call id so fresh hook processes reuse it; aliases hold no prompts, cwd, or tool arguments and are never transmitted. Missing, stale, malformed, or ambiguous state fails open to the original id, and Claude Code and Codex ids are not rewritten.
84+
- **Privacy:** Mobile Apps sends no prompt, tool input, cwd, path, URL, credential, username, hostname, Dataverse org/tenant ID, or Entra object ID. The dynamic `eventInfo` contains only `invocationSource` (`prompt` or `pretool`).
85+
- **Controls:** `scripts/lib/telemetry/ikey.json` remains `disabled: true` with a Mobile-specific placeholder until its own 1DS key, collector, stream annotation, and Kusto mapping are provisioned. That is a true hard-off: no local log and no POST. Once enabled, `/mobile-app:telemetry off` and `POWER_PLATFORM_SKILLS_TELEMETRY_MOBILE_APP_OPTOUT=1` suppress transmission while preserving the local diagnostic mirror.
86+
- **CI:** Every Mobile Apps test job must set `POWER_PLATFORM_SKILLS_TELEMETRY_MOBILE_APP_OPTOUT=1`. The single positive wire test clears that backstop only in its child process and routes the event to `POWER_PLATFORM_SKILLS_FAKE_HTTPS`; all other positive tests remain opted out and exercise the local mirror.
87+
7788
## Decisions made
7889

7990
- ✅ Markdown plan with Mermaid (no HTML rendering)

plugins/mobile-apps/README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ Example edit flows:
216216
| `/deploy` | ✅ v0 | Build + push — `npm run build` then `npx power-apps push` to the env in `power.config.json`. **Does not** drive `expo run:ios` or `expo run:android` (out of scope for v0). |
217217
| `/open-wrap-url` | ✅ v0 | Opens the Wrap URL in browser for an app ID using `https://make.powerapps.com/environments/<envID>/wrap?appID=<appID>`. Requires both `--app-id` and `--env-id`. |
218218
| `/report-issue` | ✅ v0 | Read-only diagnostic — collects env / Expo / Node versions, project context, recent errors, and renders a copy-paste-ready GitHub issue body. Sanitizes secrets. |
219+
| `/telemetry` | ✅ v0 | Enable, disable, or show the per-user Mobile Apps telemetry transmission preference. |
219220
| `/design-system` | ✅ v0 | End-to-end design system — collects brand inputs (logo, brand doc, website, free text, canvas app, code app, Figma), runs a 3-style visual picker, writes `brand/design-system.md` + `brand/tokens.ts`, renders branded screen previews. Auto-invoked at Step 6.75 of `/create-mobile-app`; also standalone. |
220221
| `/preview-screens` | ✅ v0 | Renders generated TSX screens as a browser-viewable HTML preview (no Metro needed). Uses Tamagui → HTML mapping. |
221222
| `/add-datasource` | ✅ v0 | Alias for `/add-connector` — discoverable name for "how do I connect to X?" |
@@ -237,6 +238,24 @@ Example edit flows:
237238
| `screen-builder` | Mutation — writes ONE TSX file per assigned screen, runs N in parallel |
238239
| `offline-profile-architect` | Read-only — proposes per-table row scope, relationships, selected columns, sync frequency; returns `_offline_section.md` for `/setup-offline-profile` to embed in `native-app-plan.md` |
239240
241+
## Telemetry and privacy
242+
243+
The Mobile Apps plugin includes start-only usage telemetry built on the same shared 1DS transport as Power Pages. The checked-in Mobile Apps configuration is currently `disabled: true` with a placeholder key, so it is hard-off until a dedicated Mobile Apps stream and instrumentation key are provisioned. While hard-off, it performs no telemetry shellouts, writes no local event log, and sends nothing.
244+
245+
Once provisioned, a start event can include the skill name, plugin version, session and per-start correlation IDs, OS/Node versions, AI-agent name/version, and whether the host observed the start through `UserPromptSubmit` or `PreToolUse(Skill)`. It never includes prompts, skill arguments, tool inputs, file paths, cwd, app/site names, URLs, credentials, usernames, hostnames, Dataverse organization or tenant IDs, or Entra object IDs.
246+
247+
Both host surfaces are covered — an explicit slash command and a programmatic Skill-tool call — so some hosts may produce two `skill_started` records for one visible run. The plugin does not emit `skill_completed`, success/failure, error, or duration data because the available hook boundary does not prove that the workflow itself completed.
248+
249+
Control the per-user transmission preference with:
250+
251+
```text
252+
/mobile-app:telemetry status
253+
/mobile-app:telemetry off
254+
/mobile-app:telemetry on
255+
```
256+
257+
After provisioning, `off` stops network transmission but retains the sanitized local diagnostic mirror under `~/.power-platform-skills/telemetry/mobile-app/sessions/<sessionId>/events.jsonl`. Automation can force transmission off with `POWER_PLATFORM_SKILLS_TELEMETRY_MOBILE_APP_OPTOUT=1`; this overrides the saved preference and `on`.
258+
240259
## Known blockers
241260
242261
## See also
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"_comment": "Telemetry-only start hooks. They never validate, mutate, or block tool calls. File validation stays owned by each workflow through scripts/validate-mobile-files.js.",
3+
"hooks": {
4+
"PreToolUse": [
5+
{
6+
"matcher": "Skill|skill",
7+
"hooks": [
8+
{
9+
"type": "command",
10+
"command": "node -e \"const path=require('node:path'); const root=process.env.PLUGIN_ROOT||process.env.CLAUDE_PLUGIN_ROOT; if(!root)process.exit(0); try{require(path.resolve(root,'hooks','run-telemetry.js')).start('pretool');}catch{}\"",
11+
"timeout": 10
12+
}
13+
]
14+
}
15+
],
16+
"UserPromptSubmit": [
17+
{
18+
"hooks": [
19+
{
20+
"type": "command",
21+
"command": "node -e \"const path=require('node:path'); const root=process.env.PLUGIN_ROOT||process.env.CLAUDE_PLUGIN_ROOT; if(!root)process.exit(0); try{require(path.resolve(root,'hooks','run-telemetry.js')).start('prompt');}catch{}\"",
22+
"timeout": 10
23+
}
24+
]
25+
}
26+
]
27+
}
28+
}
Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env node
2+
'use strict';
3+
4+
let telemetry;
5+
let getTrackedSkillFromPrompt;
6+
let getTrackedSkillFromToolInput;
7+
try {
8+
telemetry = require('../scripts/lib/mobile-telemetry');
9+
({
10+
getTrackedSkillFromPrompt,
11+
getTrackedSkillFromToolInput,
12+
} = require('../scripts/lib/mobileapp-hook-utils'));
13+
} catch {
14+
process.exit(0);
15+
}
16+
17+
function readStdin() {
18+
return new Promise((resolve) => {
19+
let input = '';
20+
process.stdin.setEncoding('utf8');
21+
process.stdin.on('data', (chunk) => { input += chunk; });
22+
process.stdin.on('end', () => resolve(input));
23+
process.stdin.on('error', () => resolve(input));
24+
});
25+
}
26+
27+
function invocationFor(mode, payload) {
28+
if (mode === 'prompt') return getTrackedSkillFromPrompt(payload.prompt);
29+
if (mode === 'pretool') return getTrackedSkillFromToolInput(payload.tool_input);
30+
return null;
31+
}
32+
33+
async function run(mode) {
34+
let payload;
35+
try {
36+
payload = JSON.parse(await readStdin());
37+
} catch {
38+
return;
39+
}
40+
41+
const skillName = invocationFor(mode, payload);
42+
if (!skillName) return;
43+
44+
const context = telemetry.createTelemetryContext(payload);
45+
if (context) telemetry.emitSkillStarted(context, { skillName, source: mode });
46+
}
47+
48+
function start(mode) {
49+
run(mode).catch(() => {}).finally(() => process.exit(0));
50+
}
51+
52+
if (require.main === module) start(process.argv[2]);
53+
54+
module.exports = { start };

0 commit comments

Comments
 (0)