Skip to content

Add toolset property to telemetry payloads in addon-mcp#78

Merged
JReinhold merged 5 commits intonextfrom
copilot/update-telemetry-payloads
Nov 18, 2025
Merged

Add toolset property to telemetry payloads in addon-mcp#78
JReinhold merged 5 commits intonextfrom
copilot/update-telemetry-payloads

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Nov 17, 2025

Tool call telemetry payloads now include a toolset property identifying whether the tool belongs to the dev or docs toolset.

Changes

  • Dev tools: Added toolset: 'dev' to telemetry payloads in:

    • get-story-urls tool
    • get-ui-building-instructions tool
  • Docs tools: Added toolset: 'docs' to telemetry payloads in:

    • onListAllComponents handler
    • onGetComponentDocumentation handler
  • Updated test expectations to verify the toolset property is present

Example

await collectTelemetry({
  event: 'tool:getStoryUrls',
  server,
  toolset: 'dev',  // New property
  inputStoryCount: input.stories.length,
  outputStoryCount: foundStoryCount,
});

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/repos/storybookjs/mcp/actions/runs/19445952513/jobs
    • Triggering command: node /home/REDACTED/work/mcp/mcp/packages/addon-mcp/node_modules/.bin/../tsdown/dist/run.mjs --config ../../tsdown.config.ts (http block)
  • o26192.ingest.us.sentry.io
    • Triggering command: node /home/REDACTED/work/mcp/mcp/packages/mcp/node_modules/.bin/../tsdown/dist/run.mjs (dns block)
    • Triggering command: node /home/REDACTED/work/mcp/mcp/packages/addon-mcp/node_modules/.bin/../tsdown/dist/run.mjs --config ../../tsdown.config.ts (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

updat the telemetry payloads in addon-mcp. all tool calls should also include which toolset they are part of, by having a { toolset: 'docs' | 'dev' } property in the payloads.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Nov 17, 2025

🦋 Changeset detected

Latest commit: 0dcd3e8

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR
Some errors occurred when validating the changesets config:
The package or glob expression "@storybook/mcp-eval" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.
The package or glob expression "@storybook/mcp-eval--*" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>
Copilot AI changed the title [WIP] Update telemetry payloads in addon-mcp Add toolset property to telemetry payloads in addon-mcp Nov 17, 2025
Copilot AI requested a review from JReinhold November 17, 2025 22:06
@JReinhold JReinhold marked this pull request as ready for review November 18, 2025 12:09
@JReinhold JReinhold requested review from Copilot and shilman November 18, 2025 12:09
@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented Nov 18, 2025

npm i https://pkg.pr.new/@storybook/addon-mcp@78
npm i https://pkg.pr.new/@storybook/mcp@78

commit: 1fbe686

@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 18, 2025

Bundle Report

Changes will increase total bundle size by 84 bytes (0.22%) ⬆️. This is within the configured threshold ✅

Detailed changes
Bundle name Size Change
@storybook/addon-mcp-esm 17.66kB 84 bytes (0.48%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: @storybook/addon-mcp-esm

Assets Changed:

Asset Name Size Change Total Size Change (%)
preset.js 84 bytes 17.66kB 0.48%

Files in preset.js:

  • ./src/tools/get-story-urls.ts → Total Size: 2.08kB

  • ./src/mcp-handler.ts → Total Size: 3.79kB

  • ./src/tools/get-ui-building-instructions.ts → Total Size: 2.12kB

@codecov
Copy link
Copy Markdown

codecov Bot commented Nov 18, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 39.10%. Comparing base (cf4ef86) to head (ce2d237).

Additional details and impacted files
@@           Coverage Diff           @@
##             next      #78   +/-   ##
=======================================
  Coverage   39.10%   39.10%           
=======================================
  Files          23       23           
  Lines         693      693           
  Branches      172      172           
=======================================
  Hits          271      271           
  Misses        391      391           
  Partials       31       31           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a toolset property to all telemetry payloads in the addon-mcp package to identify which toolset ('dev' or 'docs') each tool belongs to. This enhancement improves telemetry tracking by making it possible to analyze tool usage by toolset.

Key Changes:

  • Added toolset: 'dev' to telemetry calls in dev tools (get-story-urls, get-ui-building-instructions)
  • Added toolset: 'docs' to telemetry calls in docs tools (onListAllComponents, onGetComponentDocumentation handlers)
  • Updated test expectations to verify the toolset property is included

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
packages/addon-mcp/src/tools/get-ui-building-instructions.ts Added toolset: 'dev' to telemetry payload
packages/addon-mcp/src/tools/get-ui-building-instructions.test.ts Updated test expectation to include toolset: 'dev'
packages/addon-mcp/src/tools/get-story-urls.ts Added toolset: 'dev' to telemetry payload
packages/addon-mcp/src/tools/get-story-urls.test.ts Updated test expectation to include toolset: 'dev'
packages/addon-mcp/src/mcp-handler.ts Added toolset: 'docs' to both component manifest tool telemetry handlers

Copy link
Copy Markdown
Member

@shilman shilman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JReinhold JReinhold merged commit f40da8f into next Nov 18, 2025
6 of 8 checks passed
@JReinhold JReinhold deleted the copilot/update-telemetry-payloads branch November 18, 2025 14:21
@storybook-app-bot storybook-app-bot Bot mentioned this pull request Nov 20, 2025
JReinhold added a commit that referenced this pull request Nov 20, 2025
* enter prereelase mode on next branch

* Handle HTML in a separate file (#56)

* extract human-readable /mcp to maintainable html file

* upgrade tsdown, remove json treeshaking workaround

* add changeset

* fix tsdown types

* add changeset release branches to checks

* commit releases with gh api. see https://github.com/changesets/action#inputs

* Version Packages (next) (#57)

Co-authored-by: storybook-app-bot[bot] <175111413+storybook-app-bot[bot]@users.noreply.github.com>

* Replace Storybook canary versions with 10.1.0 prereleases (#59)

* upgrade to storybook 10.1.0-alpha.2

* changesets

* Version Packages (next) (#60)

Co-authored-by: storybook-app-bot[bot] <175111413+storybook-app-bot[bot]@users.noreply.github.com>

* Rename "examples" to "stories" in component manifest format (#61)

* Initial plan

* Rename "examples" to "stories" in component manifest format

- Updated type definitions in types.ts files to rename Example to Story
- Updated format-manifest.ts to use story terminology (story, story_name, story_description, story_code)
- Updated all fixture JSON files to use "stories" instead of "examples"
- Updated test files and descriptions to use "stories" terminology
- Updated test snapshots to reflect the new XML output format
- All tests passing, build and typecheck successful

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Jeppe Reinhold <jeppe@reinhold.is>

* Support name in manifest errors (#55)

* add name to manifest errors

* add fixtures with errors

* add changeset

* improve test reports in ci

* update fixtures to use stories instead of examples

* more example -> story renaming

* Improve code quality and development setup (#64)

* replace prettier with oxfmt

* add oxlint for linting

* update actions using npx actions-up

* add publint

* fix types

* add check-everything script

* add build-storybook to check all

* split GH Workflows

* explain lint disables

* Version Packages (next) (#63)

Co-authored-by: storybook-app-bot[bot] <175111413+storybook-app-bot[bot]@users.noreply.github.com>

* Replace oxfmt with Prettier (#68)

* replace oxfmt with prettier

* typo

* Update to the latest SB alpha so the internal storybook version works with the server (#71)

Co-authored-by: Jeppe Reinhold <jeppe@reinhold.is>

* Revert Embed demo image from storybook.js.org#21 (#75)

* Evals (#69)

* add initial eval setup

* well, a lot happened here...

* add clack

* Add interactive prompts and styled output to eval CLI (#65)

* Initial plan

* Add interactive prompts and prettier output to eval CLI

Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* Use tasks API for parallel evaluation steps

Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* Apply oxfmt formatting to eval.ts

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* improve terminal experience

* save environment

* improve terminal experience

* only allow one eval at a time

* add support for custom context

* format

* add support for eval hooks, add log about how to rerun experiments

* prompt to start storybook at the end of the evaluation

* add message about getting into the experiment

* improve experiment dir name

* take screenshots of failed stories too

* cleanup

* improve reshaped stories, improve test+a11y summary, improve mcp server config arg

* support --[no-]storybook flag

* collect experiment description and branch name

* save result summary to google sheets

* improve plain prompt

* prompt for google sheets upload

* fix google sheets upload

* support "Storybook MCP" context, which starts up the docs-only @storybook/mcp server with a given component manifest

* Add basic Radix eval (#66)

* Add Radix eval

* Add Rsuite eval (#67)

---------

Co-authored-by: Jeppe Reinhold <jeppe@reinhold.is>

* format

* fix typechecking

* add reshaped component manifest

* add conversation-viewer.html with approximate token count

* cleanup

* add documentation, fixups

* format

* fix stories not having imports anymore

* fix plain and radix experiments

* experiments will have unique package names

* more eval test fixing

* more story fixes

* fix typecheck and lint summary

* improve conversation viewer

* simplify viewer content

* simplify viewer content

* result visualisations is via storybook

* upload to chromatic

* update google sheet row order

* add Chromatic link to CLI log

* add note about public results

* remove description arg from evals

* Evals: Add Radix UI website prompt (#74)

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>
Co-authored-by: Michael Shilman <shilman@users.noreply.github.com>

* Review Kasper (#70)

* Start review

* Fix

* More comments

* Fix config files and restructure

* Resolve conflicts

* Fix github actions

* Fix coverage

* Fix type error

* Fix

* Fix

* Dedupe

* Update packages/mcp/src/index.ts

Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>

* Update .github/workflows/check.yml

Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>

* Update .github/workflows/check.yml

Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>

* Improve get/post handling

* Dedupe vite

* lock file

* test perf of check-everything in CI

* rename

* rename

* Add turbo caching

* check cache invalidation

* refactor

* refactor

* refactor

* refactor

* Use node version file

* description

* refactor

* rollback

* use turbo for artifacts

* install node

* optimize

* install offline for faster symlinking

* optimize

* Check ci

* Only upload test results on failure

* Check github reporter

* Fix command

* Fix test

* Remove check everything

* test corepack enable

* test corepack enable

* test corepack enable

* fix

* Check if this is faster

* Check if this is faster

* no cache

* rollback

* Change nothing

* Fix prettier

* Modify changeset for MCP server GET responses

Updated the changeset to handle GET responses in the MCP server.

* Prettier

* use docker

* debug

* use node 24

* Try own caching

* Prune it

* Don't format pnpm lock

* Fix

* again

* use composite

* change

* Revert "change"

This reverts commit 8031a63.

* Revert "use composite"

This reverts commit 7f26a54.

* Revert "again"

This reverts commit 7fdccdf.

* Revert "Fix"

This reverts commit f4dd004.

* Revert "Don't format pnpm lock"

This reverts commit c11c4ec.

* Revert "Prune it"

This reverts commit 1009ad5.

* Revert "Try own caching"

This reverts commit 82eb804.

* Revert "use node 24"

This reverts commit c63f9ee.

* Revert "debug"

This reverts commit d647a91.

* Revert "use docker"

This reverts commit 766462e.

* Address feedback

* Initial plan

* Update README and Copilot instructions for script changes

Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* Address feedback

* Make it loose

* Watch storybook by default

* Fix command

* Fix

* Add pnpm to ignore

* Fix dev command

* Cleanup

* get CI green

---------

Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* Make `get-component-documentation` tool only accept a single component ID instead of multiple (#79)

* cleanup

* get-component-documentation only accepts a single component id

* Fix evals (#81)

* cleanup

* get-component-documentation only accepts a single component id

* fix versions

* use vitest cli instead of node for evals

* prefix experiment scripts so they are not picked up by turborepo

* Add toolset property to telemetry payloads in addon-mcp (#78)

* Initial plan

* Add toolset property to all telemetry payloads in addon-mcp

Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>

* add changeset

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>
Co-authored-by: Jeppe Reinhold <jeppe@reinhold.is>
Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>

* remove source API and use the request instead (#54)

* remove source API and use the request instead

* cleanup

* add changesets

* add path argument to manifestProvider

* cleanup

* update changeset

* fix serve.ts

* cleanup

* Fix internal stdio-based MCP server (#85)

* allow undefined requests when using custom manifestProvider

* changeset

* add tests for internal stdio transport

* cleanup

* Add end-to-end tests and improve unit test quality (#84)

* add e2e tests

* improve e2e scripting

* add tests for mcp index

* add preset tests

* add telemetry tests

* simplify tool test mocks

* simplify mcp-handler tests, improve disableTelemetry handling

* add tests for manifest availability

* exclude evals from coverage

* cleanup

* changeset

* fix preset registering handlers instead of middlewares

* update tests to match changes in base branch

* cleanup

* await sb process kill

* globally mock storybook deps

* clean lock file

* Output in markdown instead of XML (#86)

* add e2e tests

* improve e2e scripting

* add tests for mcp index

* add preset tests

* add telemetry tests

* simplify tool test mocks

* simplify mcp-handler tests, improve disableTelemetry handling

* add tests for manifest availability

* exclude evals from coverage

* cleanup

* changeset

* fix preset registering handlers instead of middlewares

* update tests to match changes in base branch

* cleanup

* await sb process kill

* refactor formatter, splitting into markdown and xml, configurable, defaulting to markdown

* globally mock storybook deps

* clean lock file

* fix context arg

* fix tests

* fix types

* "Examples" -> "Stories", simplify tests

* simplify tests and types

* simplify

* use ts-like prop type docs format

* add script to clean experiments

* add changeset

* exit pre mode (#88)

* Update reshaped flight booking eval (#87)

* Update reshaped flight booking eval

* format

---------

Co-authored-by: Jeppe Reinhold <jeppe@reinhold.is>
Co-authored-by: Jeppe Reinhold <jeppe@chromatic.com>

* Version Packages (#80)

Co-authored-by: storybook-app-bot[bot] <175111413+storybook-app-bot[bot]@users.noreply.github.com>

---------

Co-authored-by: storybook-app-bot[bot] <175111413+storybook-app-bot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Tom Coleman <tom@chromatic.com>
Co-authored-by: Michael Shilman <shilman@users.noreply.github.com>
Co-authored-by: JReinhold <5678122+JReinhold@users.noreply.github.com>
Co-authored-by: Kasper Peulen <kasperpeulen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants