Tool-name drift audit: skills documenting tools their servers don't expose

[asachs01]

Fleet-wide audit of every plugin's documented MCP tool names against the tool names its shipped server actually registers. Audited against main @ 44c57a4 (post-#175), so the anti-trigger/GOVERNANCE pass (#161–#175) is included.

This issue is an audit only. No SKILL.md was edited. Corrections are a separate non-additive pass.

Headline

• 64 vendor-backed plugins audited (76 total minus 12 pack/meta plugins with no single vendor server).
• 18 have confirmed drift. 34 verified clean. 12 could not be verified.
• The "reported clean" list mostly holds — 14 of 15 confirmed; kaseya/it-glue is a false clean (1 drifted name in its GOVERNANCE.md).
• The rootly finding was reported backwards. The gateway routes rootly to mcp.rootly.com; it is the local-repo-shaped skills that are wrong, and 4 skills document names that exist on neither upstream.
• pax8's local repo is deliberate dead code, not a routing error — with a live CI hazard attached.
• The GOVERNANCE.md allowlists added in docs(email-security): anti-triggers and governance for email-security plugins #161–chore(docs): regenerate sharedSkills after shared plugin bump #175 inherited the drift. For proofpoint and knowbe4 the allowlist was transcribed from the drifted skill names, so it grants tools that do not exist while omitting real destructive ones. Proofpoint's Destructive row lists proofpoint_forensics_search_destroy (does not exist) while the real search-and-destroy tool, proofpoint_forensics_pull_messages, is absent from the allowlist entirely.

---

Summary table

Severity: critical = an agent following the primary path fails on most calls; high = a named agent/command breaks; medium = reference tables wrong, primary path works; structural = wrong server documented, not a rename.

vendor	real tools	doc'd	drifted	renamed / absent	uses	classification	severity
rootly	253 (hosted)	~60	~45	— / many	many	wrong-upstream	critical
email-security/proofpoint	44	45	31	28 / 3	105	renamed + absent	critical
email-security/knowbe4	34	36	27	22 / 5	72	renamed + absent	critical
abnormal/abnormal-security	10	31	21	18 / 3	29	renamed + absent	critical
domotz/domotz	21	38	17	15 / 2	66	renamed + absent	critical
ironscales/ironscales	9	15	6	5 / 1	51	renamed + absent	critical
sherweb/sherweb	15	21	6	4 / 2	35	renamed + absent	high
saas-alerts/saas-alerts	30	36	6	6 / 0	19	renamed	high
spamtitan/spamtitan	9	12	3	3 / 0	20	renamed	high
inforcer/inforcer	19	22	3	3 / 0	20	renamed	high
mimecast/mimecast	10	11	1	1 / 0	12	renamed	high
freshdesk/freshdesk	66	69	3	3 / 0	5	renamed	medium
threatlocker/threatlocker	17	19	2	2 / 0	2	renamed	medium
connectwise/manage	51	53	2	2 / 0	2	absent (governance)	medium
huntress/huntress	32	33	1	0 / 1	2	absent	medium
timezest/timezest	14	15	1	1 / 0	1	renamed	medium
kaseya/it-glue	24	25	1	1 / 0	1	renamed (governance)	medium
quickbooks/quickbooks-online	133	63	1	1 / 0	1	absent (governance)	medium
pax8/pax8	unverified	15	—	—	—	structural (dead local server)	high

"Uses" counts every occurrence across all doc files, not distinct names — it is the better proxy for how often an agent would actually hit a bad name.

---

The two structural cases

1. rootly — the reported finding is inverted

What is actually served. mcp-gateway/src/credentials/vendor-config.ts:2313-2318 sets containerUrl: 'https://mcp.rootly.com'. The validate() in the same entry POSTs to https://mcp.rootly.com/mcp and carries the comment "/mcp is what the gateway actually proxies against." The plugin has no .mcp.json; its README points at https://mcp.wyre.ai/v1/rootly/mcp, which proxies to Rootly. The local rootly-mcp repo is not in the request path at all.

Verified, not inferred. Unauthenticated probe → HTTP 401 (Authorization header with a valid Bearer token is required). With a real token from cortex-secret: initialize → HTTP 200, serverInfo {"name":"Rootly","version":"3.4.5"}; tools/list → 253 tools. Confirmed by dispatch, not merely by absence from a list:

tools/call	server response
incidents_get	Unknown tool: 'incidents_get'
severities_get	Unknown tool: 'severities_get'
rootly_list_postmortems	Unknown tool: 'rootly_list_postmortems'
list_severities	dispatched (upstream 404 on token scope) — not "Unknown tool"

Per-skill verdict:

skill	verdict
oncall	correct — all 5 names verified present. Ship as-is.
api-patterns	7 of 31 names real. Also claims "25 dynamically generated tools"; live count is 253.
incidents	AI/on-call names right, every CRUD name wrong.
postmortems	0 of 8 names exist on either upstream.
services	0 of 5.
alerts	0 of 4.
workflows	0 of 6.

The rootly_* names in those four skills are not the local repo's names either — the local repo uses rootly_<domain>_<verb> (rootly_alerts_list), the skills use rootly_<verb>_<noun> (rootly_list_alerts). Word order reversed. These names exist nowhere.

All 5 commands reference at least one nonexistent tool; service-status and action-items are 100% nonexistent. Both agents additionally declare tools: ["Bash","Read","Write","Glob","Grep"] in frontmatter with no mcp__* entry — even with corrected names they could not call Rootly. Separate bug.

Name mapping for the fix: incidents_get→list_incidents/search_incidents, incidents_post→create_incident, ..._alerts_get→list_incident_alerts, ..._alerts_post→attach_alert, ..._action_items_get→list_incident_action_items, ..._action_items_post→create_incident_action_item, severities_get→list_severities, services_get→list_services, teams_get→list_teams, users_get→list_users, users_me_get→get_current_user, alerts_get→list_alerts, workflows_get→list_workflows, environments_get→list_environments, functionalities_get→list_functionalities, incident_types_get→list_incident_types. The postmortems skill needs a rewrite against Rootly's retrospectives model (list_retrospective_processes, get_incident_retrospective_step, update_incident_retrospective_step, list_all_incident_action_items) — Rootly has no "postmortem" resource.

Side findings: README lists 6 skills but 7 exist (oncall, the only correct one, is missing from the table); README line 67 tells users to "Add the .mcp.json from this plugin" and no such file exists; plugin.json says 0.2.3 while the README changelog stops at 0.1.0; the gateway sets isStateful: true but the live server issues no mcp-session-id and serves tools/call without one.

2. pax8 — dead code, deliberate, with a live CI hazard

Verdict: dead code, not a routing error. The replacement was an explicit decision recorded in both repos:

• msp-claude-plugins/CHANGELOG.md:99 — "Switched to Pax8's official hosted MCP server at https://mcp.pax8.com/v1/mcp — replaces our custom pax8-mcp server"
• mcp-gateway/CHANGELOG.md:975 — "pax8: use official hosted MCP server with token auth" (6ba6c23)
• mcp-gateway/CLAUDE.md:113 — "Hosted first-party MCPs (Rootly …, PagerDuty …, Pax8, BetterStack, etc.) must NOT be in this array"

check	result
az containerapp show --name gwp-pax8 -g mcp-gateway-prod	ResourceNotFound (53 ACAs exist; gwp-huntress present as control)
azure/main.bicep vendors array	pax8 absent
docker-compose.yml (48 services)	no pax8-mcp
.harness/vendors.json (12 vendors)	pax8 absent
ghcr.io/wyre-technology/pax8-mcp	exists — 12 versions, v1.1.2 + latest, 2026-07-24
release deploy job	never run — skipped in all 18 Release runs

There is also a credential incompatibility that makes routing impossible in principle: pax8-mcp/src/index.ts:10-11 requires PAX8_CLIENT_ID/PAX8_CLIENT_SECRET (OAuth2 partner API) via X-Pax8-Client-ID/X-Pax8-Client-Secret; the gateway supplies mcpToken → x-pax8-mcp-token. Different credential, different header, different auth system. So pax8_orders_create and pax8_subscriptions_update are unreachable — not a live write exposure.

⚠️ Live hazard, worth its own issue on pax8-mcp: pax8-mcp/.github/workflows/release.yml still carries a deploy job pinned to vendor-slug: pax8 running az containerapp update --name gwp-pax8 -g mcp-gateway-prod. It has been masked purely by upstream job failures (MCP Registry + Security Scan). If those are ever fixed the deploy fires and fails on ResourceNotFound — and the obvious "fix" (creating gwp-pax8, or adding { slug: 'pax8' } to the bicep vendors array) would set VENDOR_URL_PAX8=http://gwp-pax8, clobber https://mcp.pax8.com/v1, and break the working hosted vendor. That is exactly the failure CLAUDE.md:113 documents. The repo is actively maintained (Dependabot through 2026-08-03), so this will resurface.

Docs status — cannot be closed out yet. The plugin documents 15 hyphen-delimited names (pax8-list-companies, pax8-get-company-by-uuid, …); grep "pax8_" across the whole plugin returns zero matches, so it is at least in the right namespace for the hosted server. But the "15 read-only tools" claim traces solely to skills/api-patterns/SKILL.md:18 — no Pax8-authored source confirms the count or the read-only property. Two counter-signals: the gateway's own cache allowlist (mcp-gateway/src/proxy/result-cache.ts:577-585) contains a 16th name, pax8-lookup-product, that the plugin does not list; and Pax8's OAuth metadata advertises scope Manage:Pax8Data — "Manage", not "Read". POST https://mcp.pax8.com/v1/mcp → HTTP 401 (Missing x-pax8-mcp-token header) and cortex-secret holds no Pax8 token, so the real surface stays unverified. Also: README.md:11 promises "Provision, modify, and cancel cloud subscriptions" with no write tool behind it, and both agents declare tools: ["Bash","Read","Write","Glob","Grep"] with no mcp__* entry, so neither can reach Pax8 at all.

---

Governance allowlists inherited the drift

This is the reason a governance allowlist has to be generated from tools/list rather than written from skills.

• proofpoint — GOVERNANCE.md:31 Destructive row lists proofpoint_forensics_search_destroy, which does not exist. The real destructive tool, proofpoint_forensics_pull_messages ("Auto-pull (search & destroy) messages from mailboxes"), appears in no doc file, including the allowlist. 34 of proofpoint's 105 drifted uses are inside GOVERNANCE.md.
• knowbe4 — 23 drifted uses inside GOVERNANCE.md; the entire Read row was transcribed from the drifted skill names.
• connectwise/manage, kaseya/it-glue, quickbooks — their only drift is in GOVERNANCE.md: cw_delete_time_entry, cw_update_time_entry, itglue_create_document (real name is unprefixed create_document), qbo_invoices_update. All four verified nonexistent by grep against the server sources.
• domotz is the counter-example and the model to copy — its GOVERNANCE.md:27 correctly classifies domotz_power_outlet_control as Destructive, and lines 107-108 explicitly warn that the skills "describe some tools under older names (for example domotz_list_devices or domotz_scan_network) that the current server does not expose." The governance author saw the drift and documented it rather than fixing it. (Two of domotz's GOVERNANCE.md "hits" in the table below are that warning, not new drift.)

Correction to the brief: domotz_power_outlet_control is no longer undocumented as of #167 — it is in the allowlist, correctly classified. It was undocumented at #160.

---

Undocumented tools (server exposes it, no doc mentions it)

Where a plugin has a GOVERNANCE.md the gap is now mostly closed. What remains:

plugin	undocumented	write-capable	notable
scalepad/scalepad	184 of 381	93	20+ marked ⚠ DESTRUCTIVE — IRREVERSIBLE, incl. scalepad_lm_deliverables_delete, scalepad_cm_governance_delete, scalepad_lm_assessments_delete
quickbooks/quickbooks-online	71 of 133	4	qbo_sales_receipts_create/update, qbo_vendor_credits_create/update
email-security/proofpoint	30 of 44	2	proofpoint_forensics_pull_messages — DESTRUCTIVE, removes delivered mail
email-security/knowbe4	25 of 34	1	knowbe4_execute_tool (arbitrary tool dispatch)
email-security/checkpoint-avanan	13 of 13	4	hec_delete_exception — DESTRUCTIVE, IRREVERSIBLE; plus hec_restore_emails, hec_restore_events, hec_update_exception. Plugin has no GOVERNANCE.md and names no tools anywhere.
azure-mcp/azure-mcp	8 of 11	0	advisor, applens, foundry, monitor, pricing, quota, resourcehealth, search
m365/m365	3 of 3	0	all three microsoft_graph_* tools
sentinelone/sentinelone	1 of 22	0	powerquery

email-security/checkpoint-avanan is the highest-risk remaining gap: an allowlist built from its docs today would contain nothing, while the server ships an irreversible delete.

---

Per-vendor detail

Nearest-real-name is a similarity heuristic (segment Jaccard + Levenshtein). It is a starting point, not an authority — e.g. domotz_list_alerts is matched to domotz_alerts_profiles_list where domotz_alerts_device_list is more likely correct. Verify each before applying. — no equivalent exists means nothing scored above threshold: the documented capability is absent from the server, so the fix is to delete the claim, not rename it.

Expand all 17 script-detected vendors

abnormal/abnormal-security — server abnormal-mcp, 10 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
abnormal_list_cases	abnormal_cases_list	renamed	3	agents/email-threat-analyst.md:20
abnormal_list_threats	abnormal_threats_list	renamed	3	agents/email-threat-analyst.md:18
abnormal_cases_action	abnormal_cases_get	renamed	2	GOVERNANCE.md:95
abnormal_manage_remediation	abnormal_remediation_manage	renamed	2	agents/email-threat-analyst.md:18
abnormal_messages_headers	abnormal_messages_get	renamed	2	GOVERNANCE.md:96
abnormal_threats_remediate	abnormal_threats_get	renamed	2	GOVERNANCE.md:95
abnormal_ato_activity	— no equivalent exists	absent	1	skills/account-takeover/SKILL.md:92
abnormal_ato_cases_get	abnormal_cases_get	renamed	1	skills/account-takeover/SKILL.md:91
abnormal_ato_cases_list	abnormal_cases_list	renamed	1	skills/account-takeover/SKILL.md:90
abnormal_ato_remediate	— no equivalent exists	absent	1	skills/account-takeover/SKILL.md:93
abnormal_cases_actions	abnormal_cases_get	renamed	1	skills/cases/SKILL.md:113
abnormal_get_abuse_reports	abnormal_abuse_list	renamed	1	agents/email-threat-analyst.md:20
abnormal_get_message	abnormal_messages_get	renamed	1	agents/email-threat-analyst.md:18
abnormal_get_threat	abnormal_cases_get	renamed	1	agents/email-threat-analyst.md:18
abnormal_list_messages	abnormal_messages_list	renamed	1	agents/email-threat-analyst.md:18
abnormal_threats_actions	abnormal_threats_get	renamed	1	skills/threats/SKILL.md:120
abnormal_threats_unremediate	abnormal_threats_get	renamed	1	skills/threats/SKILL.md:122
abnormal_vendors_activity	— no equivalent exists	absent	1	skills/vendors/SKILL.md:93
abnormal_vendors_get	abnormal_messages_get	renamed	1	skills/vendors/SKILL.md:92
abnormal_vendors_list	abnormal_messages_list	renamed	1	skills/vendors/SKILL.md:91
abnormal_vendors_threats	abnormal_threats_get	renamed	1	skills/vendors/SKILL.md:94

connectwise/manage — server connectwise-manage-mcp, 51 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
cw_delete_time_entry	cw_create_time_entry	renamed	1	GOVERNANCE.md:49
cw_update_time_entry	cw_create_time_entry	renamed	1	GOVERNANCE.md:50

domotz/domotz — server domotz-mcp, 21 real tools (source: static+runtime+live-gw)

documented (wrong)	real name	class	uses	first site
domotz_list_devices	domotz_devices_list	renamed	11	GOVERNANCE.md:107
domotz_list_agents	domotz_agents_list	renamed	9	commands/alert-status.md:14
domotz_get_agent	domotz_agents_get	renamed	7	commands/device-inventory.md:14
domotz_list_alerts	domotz_alerts_profiles_list	renamed	6	commands/alert-status.md:14
domotz_scan_network	domotz_network_interfaces	renamed	5	GOVERNANCE.md:108
domotz_search_devices	domotz_devices_get	renamed	5	commands/device-lookup.md:14
domotz_get_device	domotz_devices_get	renamed	4	commands/device-inventory.md:14
domotz_list_eyes	domotz_agents_list	renamed	4	commands/site-overview.md:14
domotz_get_alert	domotz_agents_get	renamed	2	commands/alert-status.md:14
domotz_list_alert_profiles	domotz_alerts_profiles_list	renamed	2	skills/alerts/SKILL.md:118
domotz_list_eye_results	— no equivalent exists	absent	2	skills/api-patterns/SKILL.md:95
domotz_list_ports	domotz_agents_list	renamed	2	skills/api-patterns/SKILL.md:86
domotz_list_snmp_data	domotz_metrics_snmp_sensors_list	renamed	2	skills/api-patterns/SKILL.md:85
domotz_run_speed_test	— no equivalent exists	absent	2	skills/api-patterns/SKILL.md:87
domotz_get_agent_status	domotz_status	renamed	1	skills/api-patterns/SKILL.md:61
domotz_get_device_status	domotz_status	renamed	1	skills/api-patterns/SKILL.md:70
domotz_get_eye	domotz_agents_get	renamed	1	skills/api-patterns/SKILL.md:94

email-security/knowbe4 — server knowbe4-mcp, 34 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
knowbe4_training_list_campaigns	knowbe4_training_campaigns_list	renamed	6	GOVERNANCE.md:28
knowbe4_training_list_enrollments	knowbe4_training_enrollments_list	renamed	6	GOVERNANCE.md:28
knowbe4_phishing_list_campaigns	knowbe4_phishing_campaigns_list	renamed	4	GOVERNANCE.md:28
knowbe4_training_get_campaign	knowbe4_training_campaigns_get	renamed	4	GOVERNANCE.md:28
knowbe4_groups_list_members	knowbe4_groups_members	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_get_campaign	knowbe4_phishing_campaigns_get	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_get_recipient	knowbe4_phishing_campaigns_get	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_get_security_test	knowbe4_phishing_security_test_get	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_list_recipients	knowbe4_phishing_campaigns_list	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_list_security_tests	knowbe4_phishing_security_tests_list	renamed	3	GOVERNANCE.md:28
knowbe4_training_list_modules	knowbe4_training_campaigns_list	renamed	3	GOVERNANCE.md:28
knowbe4_training_list_users	knowbe4_users_list	renamed	3	agents/security-awareness-analyst.md:20
knowbe4_users_list_events	knowbe4_users_list	renamed	3	GOVERNANCE.md:28
knowbe4_phishing_get_template	knowbe4_phishing_campaigns_get	renamed	2	GOVERNANCE.md:28
knowbe4_phishing_list_templates	knowbe4_phishing_campaigns_list	renamed	2	GOVERNANCE.md:28
knowbe4_reporting_account_summary	knowbe4_reporting_phishing_summary	renamed	2	GOVERNANCE.md:28
knowbe4_reporting_department_breakdown	— no equivalent exists	absent	2	GOVERNANCE.md:28
knowbe4_reporting_ppp_trend	— no equivalent exists	absent	2	GOVERNANCE.md:28
knowbe4_training_get_enrollment	knowbe4_training_enrollments_get	renamed	2	GOVERNANCE.md:28
knowbe4_training_get_module	knowbe4_training_campaigns_get	renamed	2	GOVERNANCE.md:28
knowbe4_training_get_store_purchase	knowbe4_training_enrollments_get	renamed	2	GOVERNANCE.md:28
knowbe4_training_list_phishing_tests	knowbe4_phishing_security_tests_list	renamed	2	agents/training-enforcer.md:22
knowbe4_training_list_store_purchases	knowbe4_store_purchases_list	renamed	2	GOVERNANCE.md:28
knowbe4_training_list_training_campaigns	knowbe4_training_campaigns_list	renamed	2	agents/training-enforcer.md:20
knowbe4_phisher_bulk_action	— no equivalent exists	absent	1	agents/security-awareness-analyst.md:18
knowbe4_phisher_get_message	— no equivalent exists	absent	1	agents/security-awareness-analyst.md:18
knowbe4_phisher_list_messages	— no equivalent exists	absent	1	agents/security-awareness-analyst.md:18

email-security/proofpoint — server proofpoint-mcp, 44 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
proofpoint_forensics_search_destroy	proofpoint_forensics_search_messages	renamed	6	GOVERNANCE.md:31
proofpoint_forensics_get_evidence	proofpoint_forensics_get_threat	renamed	5	GOVERNANCE.md:29
proofpoint_forensics_get_report	proofpoint_forensics_get_threat	renamed	5	GOVERNANCE.md:29
proofpoint_quarantine_bulk_release	proofpoint_quarantine_release	renamed	5	GOVERNANCE.md:31
proofpoint_quarantine_preview	proofpoint_quarantine_delete	renamed	5	GOVERNANCE.md:29
proofpoint_tap_get_all_events	proofpoint_tap_get_all_threats	renamed	5	GOVERNANCE.md:29
proofpoint_threat_get_indicators	proofpoint_threat_get_iocs	renamed	5	GOVERNANCE.md:29
proofpoint_url_get_clicks	proofpoint_tap_get_clicks_blocked	renamed	5	GOVERNANCE.md:29
proofpoint_threat_search_campaigns	proofpoint_threat_get_campaign	renamed	4	GOVERNANCE.md:29
proofpoint_url_batch_decode	proofpoint_url_decode	renamed	4	GOVERNANCE.md:29
proofpoint_forensics_auto_pull_status	proofpoint_forensics_pull_messages	renamed	3	GOVERNANCE.md:29
proofpoint_forensics_get_operation	proofpoint_forensics_get_campaign	renamed	3	GOVERNANCE.md:29
proofpoint_forensics_get_sandbox_report	proofpoint_forensics_get_threat	renamed	3	GOVERNANCE.md:29
proofpoint_forensics_message_trace	— no equivalent exists	absent	3	GOVERNANCE.md:29
proofpoint_get_campaign	proofpoint_threat_get_campaign	renamed	3	agents/email-security-auditor.md:19
proofpoint_get_siem_clicks	proofpoint_tap_get_clicks_blocked	renamed	3	agents/email-security-auditor.md:19
proofpoint_get_siem_messages	proofpoint_tap_get_messages_blocked	renamed	3	agents/email-security-auditor.md:19
proofpoint_people_get_attack_index	proofpoint_people_get_vap	renamed	3	GOVERNANCE.md:29
proofpoint_people_list_vip	proofpoint_people_get_vap	renamed	3	GOVERNANCE.md:29
proofpoint_quarantine_bulk_delete	proofpoint_quarantine_delete	renamed	3	GOVERNANCE.md:31
proofpoint_threat_get_family	proofpoint_threat_get_campaign	renamed	3	GOVERNANCE.md:29
proofpoint_threat_get_landscape	proofpoint_threat_get_campaign	renamed	3	GOVERNANCE.md:29
proofpoint_threat_search_indicators	— no equivalent exists	absent	3	GOVERNANCE.md:29
proofpoint_url_get_verdict	proofpoint_url_decode	renamed	3	GOVERNANCE.md:29
proofpoint_forensics_list_operations	— no equivalent exists	absent	2	GOVERNANCE.md:29
proofpoint_get_email_stats	proofpoint_events_get_stats	renamed	2	agents/email-security-auditor.md:21
proofpoint_list_orgs	proofpoint_list_categories	renamed	2	agents/email-security-auditor.md:21
proofpoint_people_set_vip	proofpoint_people_get_vap	renamed	2	GOVERNANCE.md:30
proofpoint_quarantine_get	proofpoint_quarantine_list	renamed	2	GOVERNANCE.md:29
proofpoint_tap_get_top_clickers	proofpoint_people_get_top_clickers	renamed	2	GOVERNANCE.md:29
proofpoint_threat_get_actor	proofpoint_threat_get_iocs	renamed	2	GOVERNANCE.md:29

freshdesk/freshdesk — server freshdesk-mcp, 66 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
freshdesk_sla_policies_list	freshdesk_sla_list	renamed	2	README.md:76
freshdesk_tickets_conversations	freshdesk_tickets_list_conversations	renamed	2	README.md:59
freshdesk_solutions_articles_search	freshdesk_solutions_articles_create	renamed	1	README.md:73

huntress/huntress — server huntress-mcp, 32 real tools (source: static+runtime+prod-schema)

documented (wrong)	real name	class	uses	first site
huntress_back	— no equivalent exists	absent	2	skills/api-patterns/SKILL.md:48

inforcer/inforcer — server inforcer-mcp, 19 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
inforcer_secure_scores	inforcer_secure_scores_get	renamed	10	GOVERNANCE.md:51
inforcer_audit_events_search	inforcer_audit_search	renamed	6	GOVERNANCE.md:51
inforcer_tenant_policies_list	inforcer_policies_list	renamed	4	GOVERNANCE.md:52

ironscales/ironscales — server ironscales-mcp, 9 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
ironscales_classify_email	ironscales_email_classify	renamed	11	agents/phishing-responder.md:19
ironscales_get_company_stats	ironscales_stats_company	renamed	10	agents/crowdsourced-intel-harvester.md:21
ironscales_get_incident	ironscales_incidents_get	renamed	10	agents/phishing-responder.md:19
ironscales_list_incidents	ironscales_incidents_list	renamed	8	agents/phishing-responder.md:36
ironscales_remediate_incident	— no equivalent exists	absent	7	agents/phishing-responder.md:19
ironscales_manage_allowlist	ironscales_allowlist_manage	renamed	5	agents/phishing-responder.md:21

kaseya/it-glue — server itglue-mcp, 24 real tools (source: static+prod-schema+live-gw)

documented (wrong)	real name	class	uses	first site
itglue_create_document	create_document	renamed	1	GOVERNANCE.md:81

mimecast/mimecast — server mimecast-mcp, 10 real tools (source: static)

documented (wrong)	real name	class	uses	first site
mimecast_get_queue	mimecast_get_queue_status	renamed	12	GOVERNANCE.md:113

quickbooks/quickbooks-online — server qbo-mcp, 133 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
qbo_invoices_update	qbo_invoices_create	renamed	1	GOVERNANCE.md:130

saas-alerts/saas-alerts — server saas-alerts-mcp, 30 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
saas_alerts_users_get	saas_alerts_users_get_msp	renamed	7	README.md:54
saas_alerts_devices_get	saas_alerts_customers_get	renamed	4	README.md:57
saas_alerts_billing_get	saas_alerts_billing_get_details	renamed	2	README.md:60
saas_alerts_billing_list	saas_alerts_billing_list_dates	renamed	2	README.md:60
saas_alerts_devices_list	saas_alerts_devices_list_orgs	renamed	2	README.md:57
saas_alerts_users_list	saas_alerts_users_list_partner	renamed	2	README.md:54

sherweb/sherweb — server sherweb-mcp, 15 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
sherweb_billing_get_billing_periods	— no equivalent exists	absent	9	commands/billing-summary.md:14
sherweb_billing_get_payable_charges	sherweb_billing_payable_charges	renamed	8	commands/billing-summary.md:14
sherweb_customers_get_accounts_receivable	sherweb_customers_accounts_receivable	renamed	8	commands/list-customers.md:14
sherweb_billing_get_invoices	— no equivalent exists	absent	5	skills/api-patterns/SKILL.md:177
sherweb_billing_get_invoice_details	sherweb_billing_charge_details	renamed	3	skills/api-patterns/SKILL.md:178
sherweb_billing_get_charge_details	sherweb_billing_charge_details	renamed	2	skills/api-patterns/SKILL.md:176

spamtitan/spamtitan — server spamtitan-mcp, 9 real tools (source: static+runtime)

documented (wrong)	real name	class	uses	first site
spamtitan_list_allowlist	spamtitan_manage_allowlist	renamed	9	GOVERNANCE.md:126
spamtitan_list_blocklist	spamtitan_manage_blocklist	renamed	8	GOVERNANCE.md:126
spamtitan_get_domain_stats	spamtitan_get_stats	renamed	3	GOVERNANCE.md:127

threatlocker/threatlocker — server threatlocker-mcp, 17 real tools (source: static+prod-schema)

documented (wrong)	real name	class	uses	first site
threatlocker_computer_groups_get	threatlocker_computer_groups_list	renamed	1	README.md:60
threatlocker_organizations_get	threatlocker_organizations_get_auth_key	renamed	1	README.md:73

timezest/timezest — server timezest-mcp, 14 real tools (source: static+live-gw)

documented (wrong)	real name	class	uses	first site
timezest_resources_get	timezest_resources_list	renamed	1	skills/resources/SKILL.md:62

---

Recommended fix order

Ordered by agent-breakage, not by count.

1. rootly — 4 skills (postmortems, services, alerts, workflows) document a server that does not exist. Delete or rewrite; do not rename. Then fix api-patterns (24 of 31 names fabricated, "25 tools" → 253) and incidents. Leave oncall alone. Add mcp__* entries to both agents' frontmatter.
2. email-security/proofpoint then email-security/knowbe4 — highest absolute breakage (105 and 72 uses), and their GOVERNANCE allowlists must be regenerated from tools/list, not patched by hand. Proofpoint first: its Destructive row is wrong in both directions.
3. abnormal/abnormal-security — 21 of 31 documented names wrong against a 10-tool server. Whole skills (vendors, account-takeover) describe capability the server does not have.
4. domotz/domotz — 100% of skill and command names wrong, 66 uses. Governance is already correct, so this is a mechanical skills-and-commands rename.
5. ironscales/ironscales — 51 uses against a 9-tool server; every name in agents/phishing-responder.md is wrong.
6. sherweb, saas-alerts, spamtitan, inforcer, mimecast — contained renames, 12–35 uses each.
7. freshdesk, threatlocker, timezest, huntress — reference-table and README fixes; primary paths work.
8. connectwise/manage, kaseya/it-glue, quickbooks — one-line GOVERNANCE.md corrections.
9. email-security/checkpoint-avanan — not drift, but the largest governance gap: 13 tools, 4 write-capable, 1 irreversible, zero documented.
10. pax8 — docs unblocked only after someone generates an x-pax8-mcp-token and captures the real tools/list. Separately, file the release.yml deploy-job hazard against pax8-mcp.

---

Confirming / refuting the prior reports

Reported drifted (14): confirmed for inforcer, saas-alerts, threatlocker, abnormal, ironscales, mimecast, spamtitan, domotz, freshdesk, sherweb, rootly — 11. rootly confirmed but inverted, see above. betterstack and pagerduty could not be verified (hosted, no credentials). pax8 is structural rather than name-drift, and its hosted surface is unverified.

Newly found, not previously reported (7): email-security/proofpoint, email-security/knowbe4, huntress, timezest, connectwise/manage, kaseya/it-glue, quickbooks/quickbooks-online.

Reported clean (15): confirmed for kaseya/autotask (98 tools, 98 names, 0 drift), datto-bcdr, datto-rmm, datto-saas-protection, kaseya-bms, kaseya-vsa, rocketcyber, spanning, unitrends, connectwise-cpq, auvik, meraki, cipp, blumira — 14 of 15. Refuted: kaseya/it-glue — GOVERNANCE.md:81 says itglue_create_document; the server registers create_document (unprefixed). One name, but it is a false clean.

Also verified clean beyond the reported list: alternative-payments, atera, connectwise/automate, crewhu, halopsa, hudu, kaseya-quote-manager, liongard, microsoft-graph, ncentral, ninjaone-rmm, salesbuildr, scalepad, sentinelone, superops-ai, syncro-msp, xero.

---

Unverified — explicitly NOT called clean

Twelve plugins where ground truth could not be established. All are hosted upstreams or vendors with no local server repo. None should be treated as clean.

plugin	upstream	why unverified
pax8/pax8	https://mcp.pax8.com/v1	401 Missing x-pax8-mcp-token; no token in cortex-secret
pagerduty/pagerduty	https://mcp.pagerduty.com	401 on unauthenticated initialize; documents 8 pagerduty_* names
betterstack/betterstack	https://mcp.betterstack.com	404 on unauthenticated POST /mcp — worth a separate look, since the gateway's validate() uses that exact path; documents 5 betterstack_* names
hubspot/hubspot	https://mcp.hubspot.com/anthropic/v1/mcp	OAuth-gated; documents 26 names
slack/slack	https://mcp.slack.com	OAuth-gated; documents 0 tool names
stripe/stripe	https://mcp.stripe.com	OAuth-gated; documents 0 tool names
pandadoc/pandadoc	https://mcp.pandadoc.com/v1/mcp	OAuth-gated; documents 0 tool names
warmly/warmly	https://opps-api.getwarmly.com/api/mcp	auth-gated; documents 0 tool names
runzero/runzero	https://console.runzero.com/mcp	auth-gated; documents 19 runzero_* names
blackpoint/blackpoint	http://blackpoint-mcp	sidecar in vendor-config, no local repo available; documents 15 names
immybot/immybot	http://immybot-mcp	sidecar in vendor-config, no local repo available; documents 68 names
clio/clio	—	not in vendor-config.ts at all; documents 26 clio_* names

blackpoint, immybot and clio are the cheapest to resolve (check out the repo / add the vendor entry), and immybot documents 68 tool names with nothing currently verifying them.

Also flagged, not drift: mimecast, spamtitan, ironscales, sherweb, scalepad and connectwise-cpq ship plugins but have no entry in mcp-gateway/vendor-config.ts, so gateway users cannot reach them at all regardless of tool names.

---

Method

1. Ground truth, union of four sources per vendor:
   • runtime — spawn each <vendor>-mcp/dist/index.js over stdio, initialize → tools/list. 50 of 61 repos answered. Authoritative but incomplete for progressive-disclosure servers: 12 gate their surface behind a *_navigate call and advertise only 1–2 entrypoints at init (pax8-mcp returns exactly pax8_navigate).
   • static — scan src/** for MCP Tool object literals (name: "…" with an adjacent inputSchema) and server.registerTool(…) calls. Recovers the post-navigation surface. prompts.ts is excluded — MCP prompt argument definitions have the identical { name: '…' } shape and produced false tools (client_name, incident_id, customer_id) on the first pass.
   • prod-schema — mcp-gateway/schemas/<vendor>.json, pinned to the prod image digests in .harness/vendors.json. 12 vendors.
   • live-gw — tool names observed on the live mcp.wyre.ai connector during the audit. This is what resolved sentinelone (22 tools, no vendor prefix — list_alerts, powerquery, purple_ai), which both other sources missed.
   • Cross-check: the script reports staticMissed (runtime tools the static scan did not find). qbo-mcp and scalepad-mcp register from generated registries the static scan cannot see — for those two, runtime is authoritative and static alone would badly under-report.
2. Documented names — backtick-quoted snake_case tokens from each plugin's skills/**/SKILL.md, references/*.md, agents/*.md, commands/*.md, and root-level README.md / GOVERNANCE.md, filtered to tokens whose first segment matches a prefix the server actually uses.
3. Diff — set difference, plus a similarity score to split renamed (near-match exists) from absent (nothing close).
4. Hosted vendors — vendor-config.ts containerUrl decides the real upstream. Where that is a vendor host, the local repo is ignored entirely; that is how the rootly and pax8 determinations were reached.

Verified vs inferred. Everything in the summary table and per-vendor detail is verified against server source or a live tools/list. Rootly's 253-tool surface was verified with a real token and confirmed per-name by tools/call dispatch. Pax8's hosted surface is inferred only — the "15 read-only tools" claim has no vendor-authored source. Everything in the Unverified section is explicitly not established.

Known limits of the script. It only sees names written in backticks; prose mentions are missed. The nearest-name suggestion is a heuristic. It cannot reach OAuth-gated hosted upstreams. Servers with no dist/ and no recognizable registration pattern fall through (3cx-mcp, float-mcp, nable-mcp, pco-mcp, autotask-projects-mcp).

The script

Two files. probe-tools.mjs builds the runtime source; tool-drift-audit.mjs does the diff and exits 1 on drift, so it can gate CI. A separate PR proposes adding it to scripts/ alongside check-marketplace-drift.mjs.

node scripts/probe-tools.mjs /path/to/mcp-servers ground-truth.json
node scripts/tool-drift-audit.mjs \
  --plugins . --servers /path/to/mcp-servers --gateway /path/to/mcp-gateway \
  --runtime ground-truth.json --live live-gw.json --json audit.json

scripts/tool-drift-audit.mjs

#!/usr/bin/env node
/**
 * tool-drift-audit.mjs — cross-check every plugin's documented MCP tool names
 * against the tool names its shipped server actually registers.
 *
 * Ground truth is the union of up to four complementary sources:
 *   1. RUNTIME     — spawn `node dist/index.js` over stdio, initialize, tools/list
 *                    (see probe-tools.mjs). Authoritative, but progressive-disclosure
 *                    ("navigate") servers advertise only 1-2 entrypoints before
 *                    navigation, so it under-reports those.
 *   2. STATIC      — scan src/ for MCP `Tool` object literals (`name: "..."` with an
 *                    adjacent `inputSchema`) and `server.registerTool(...)` calls.
 *                    Covers the post-navigation surface.
 *   3. PROD-SCHEMA — mcp-gateway/schemas/<vendor>.json, pinned to the prod image
 *                    digests in .harness/vendors.json. 12 vendors only.
 *   4. LIVE-GW     — tool names observed on the live gateway connector, supplied
 *                    via --live <json> as { vendorId: [names] }.
 *
 * Documented names are the backtick-quoted snake_case tokens in each plugin's
 * skills/**\/SKILL.md, references/*.md, agents/*.md, commands/*.md and its
 * root-level docs (README.md, GOVERNANCE.md), filtered to tokens whose first
 * segment matches a prefix the server actually uses.
 *
 * Exit code 1 when any drift is found, so it can gate CI.
 *
 * Usage:
 *   node tool-drift-audit.mjs --plugins <msp-claude-plugins-repo-root> \
 *                             --servers <dir containing the *-mcp repos> \
 *                             --gateway <mcp-gateway checkout> \
 *                             [--runtime probe.json] [--live live.json] [--json out.json]
 */
import { readdirSync, readFileSync, existsSync, writeFileSync } from "node:fs";
import { join, relative, extname } from "node:path";

// ---------------------------------------------------------------- args
const argv = process.argv.slice(2);
const arg = (n, d) => { const i = argv.indexOf(n); return i >= 0 ? argv[i + 1] : d; };
const PLUGINS_REPO = arg("--plugins");
const SERVERS_ROOT = arg("--servers", "/Users/asachs/mcp");
const GATEWAY = arg("--gateway", "/Users/asachs/mcp/mcp-gateway");
const JSON_OUT = arg("--json");
const RUNTIME_JSON = arg("--runtime"); // optional pre-computed probe output
const LIVE_JSON = arg("--live");       // optional captured live-gateway tool lists
if (!PLUGINS_REPO) { console.error("--plugins <repo root> required"); process.exit(2); }
const PLUGINS = join(PLUGINS_REPO, "msp-claude-plugins");

// ---------------------------------------------------------------- plugin -> server/vendor map
// Plugin dirs whose slug does not equal `<server-repo minus -mcp>` are listed here.
// `server: null` = no local WYRE server repo (hosted vendor or unbuilt).
const PLUGIN_MAP = {
  "abnormal/abnormal-security": { server: "abnormal-mcp", vendor: "abnormal-security" },
  "connectwise/automate": { server: "connectwise-automate-mcp", vendor: "connectwise-automate" },
  "connectwise/cpq": { server: "connectwise-cpq-mcp", vendor: null },
  "connectwise/manage": { server: "connectwise-manage-mcp", vendor: "connectwise-manage" },
  "email-security/checkpoint-avanan": { server: "avanan-mcp", vendor: "avanan" },
  "email-security/knowbe4": { server: "knowbe4-mcp", vendor: "knowbe4" },
  "email-security/proofpoint": { server: "proofpoint-mcp", vendor: "proofpoint" },
  "kaseya/autotask": { server: "autotask-mcp", vendor: "autotask" },
  "kaseya/datto-bcdr": { server: "datto-bcdr-mcp", vendor: "datto-bcdr" },
  "kaseya/datto-rmm": { server: "datto-rmm-mcp", vendor: "datto-rmm" },
  "kaseya/datto-saas-protection": { server: "datto-saas-protection-mcp", vendor: "datto-saas-protection" },
  "kaseya/it-glue": { server: "itglue-mcp", vendor: "itglue" },
  "kaseya/kaseya-bms": { server: "kaseya-bms-mcp", vendor: "kaseya-bms" },
  "kaseya/kaseya-vsa": { server: "kaseya-vsa-mcp", vendor: "kaseya-vsa" },
  "kaseya/rocketcyber": { server: "rocketcyber-mcp", vendor: "rocketcyber" },
  "kaseya/spanning": { server: "spanning-mcp", vendor: "spanning" },
  "kaseya/unitrends": { server: "unitrends-mcp", vendor: "unitrends" },
  "ninjaone/ninjaone-rmm": { server: "ninjaone-mcp", vendor: "ninjaone" },
  "quickbooks/quickbooks-online": { server: "qbo-mcp", vendor: "qbo" },
  "superops/superops-ai": { server: "superops-mcp", vendor: "superops" },
  "syncro/syncro-msp": { server: "syncro-mcp", vendor: "syncro" },
  // Hosted upstreams — the local repo (if any) is NOT what the gateway serves.
  "pax8/pax8": { server: null, vendor: "pax8", hostedNote: "local pax8-mcp exists but gateway routes to https://mcp.pax8.com/v1" },
  "rootly/rootly": { server: null, vendor: "rootly", hostedNote: "local rootly-mcp exists but gateway routes to https://mcp.rootly.com" },
  "pagerduty/pagerduty": { server: null, vendor: "pagerduty" },
  "betterstack/betterstack": { server: null, vendor: "betterstack" },
  "hubspot/hubspot": { server: null, vendor: "hubspot" },
  "pandadoc/pandadoc": { server: null, vendor: "pandadoc" },
  "runzero/runzero": { server: null, vendor: "runzero" },
  "slack/slack": { server: null, vendor: "slack" },
  "stripe/stripe": { server: null, vendor: "stripe" },
  "warmly/warmly": { server: null, vendor: "warmly" },
  "microsoft-graph/microsoft-graph": { server: null, vendor: "microsoft-graph" },
  "m365/m365": { server: null, vendor: "microsoft-graph" },
  "azure-mcp/azure-mcp": { server: null, vendor: "azure-mcp" },
  "clio/clio": { server: null, vendor: null },
  "blackpoint/blackpoint": { server: null, vendor: "blackpoint" },
  "immybot/immybot": { server: null, vendor: "immybot" },
};
// Aggregate/meta plugins: no single vendor server.
const SKIP = new Set(["shared", "wyre-gateway", "assets-pack", "awareness-pack", "backup-pack",
  "cloudops-pack", "compliance-pack", "devops-pack", "finance-pack", "ops-pack",
  "sales-pack", "secops-pack"]);

// ---------------------------------------------------------------- discovery
function findPluginDirs(base) {
  const found = [];
  const walk = (dir, depth) => {
    if (depth > 3) return;
    let entries; try { entries = readdirSync(dir, { withFileTypes: true }); } catch { return; }
    for (const e of entries) {
      if (!e.isDirectory() || e.name === "node_modules" || e.name === ".git") continue;
      const p = join(dir, e.name);
      if (e.name === ".claude-plugin" && existsSync(join(p, "plugin.json"))) { found.push(dir); continue; }
      walk(p, depth + 1);
    }
  };
  walk(base, 0);
  return found.sort();
}

function walkFiles(dir, pred, acc = []) {
  let entries; try { entries = readdirSync(dir, { withFileTypes: true }); } catch { return acc; }
  for (const e of entries) {
    const p = join(dir, e.name);
    if (e.isDirectory()) {
      if (["node_modules", "__tests__", "dist", ".git"].includes(e.name)) continue;
      walkFiles(p, pred, acc);
    } else if (pred(p, e.name)) acc.push(p);
  }
  return acc;
}

// ---------------------------------------------------------------- ground truth: static
const TOOLISH = /^[a-z][a-z0-9]*(?:_[a-z0-9]+)+$/;
function staticTools(serverDir) {
  const src = join(SERVERS_ROOT, serverDir, "src");
  if (!existsSync(src)) return null;
  const files = walkFiles(src, (p, n) =>
    [".ts", ".js", ".mts", ".mjs"].includes(extname(n)) &&
    !/\.(test|spec)\.[cm]?[tj]s$/.test(n) && !/\.d\.ts$/.test(n) &&
    // prompts.ts declares MCP *prompt* arguments as `{ name: 'client_name' }` —
    // same object shape as a Tool literal, but not a tool.
    !/^prompts\.[cm]?[tj]s$/.test(n));
  const tools = new Set();
  const REG = /\.(?:registerTool|tool|addTool)\(\s*(['"`])([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\1/g;
  for (const f of files) {
    const text = readFileSync(f, "utf8");
    const lines = text.split("\n");
    // MCP `Tool` object literals: `name: "x"` with an adjacent inputSchema/description.
    for (let i = 0; i < lines.length; i++) {
      const m = lines[i].match(/name:\s*(['"`])([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\1/);
      if (!m) continue;
      const win = lines.slice(Math.max(0, i - 3), i + 16).join("\n");
      if (/inputSchema/.test(win)) tools.add(m[2]);
    }
    // Imperative registration: server.registerTool("x", ...) — often multi-line.
    let r; REG.lastIndex = 0;
    while ((r = REG.exec(text))) tools.add(r[2]);
  }
  return [...tools].sort();
}

// ---------------------------------------------------------------- documented names
const DOC_GLOBS = (dir) => walkFiles(dir, (p, n) => {
  if (extname(n) !== ".md") return false;
  const rel = relative(dir, p);
  // Plugin-root docs (README.md, GOVERNANCE.md, …) plus the four content dirs.
  return !rel.includes("/") || /^skills\//.test(rel) || /^references\//.test(rel) ||
         /^agents\//.test(rel) || /^commands\//.test(rel);
});

function documentedNames(pluginDir) {
  const files = DOC_GLOBS(pluginDir);
  const hits = new Map(); // name -> [file:line]
  for (const f of files) {
    const lines = readFileSync(f, "utf8").split("\n");
    for (let i = 0; i < lines.length; i++) {
      // backtick-quoted tokens, plus mcp__vendor__tool references
      for (const m of lines[i].matchAll(/`([^`\s]+)`/g)) {
        let t = m[1];
        const ns = t.match(/^mcp__[a-z0-9-]+__(.+)$/);
        if (ns) t = ns[1];
        if (!TOOLISH.test(t)) continue;
        if (!hits.has(t)) hits.set(t, []);
        hits.get(t).push(`${relative(pluginDir, f)}:${i + 1}`);
      }
    }
  }
  return hits;
}

// ---------------------------------------------------------------- classification
function prefixesOf(tools) {
  const p = new Set();
  for (const t of tools) p.add(t.split("_")[0]);
  return p;
}
function levenshtein(a, b) {
  const m = a.length, n = b.length;
  const d = Array.from({ length: m + 1 }, (_, i) => [i, ...Array(n).fill(0)]);
  for (let j = 0; j <= n; j++) d[0][j] = j;
  for (let i = 1; i <= m; i++) for (let j = 1; j <= n; j++)
    d[i][j] = Math.min(d[i-1][j] + 1, d[i][j-1] + 1, d[i-1][j-1] + (a[i-1] === b[j-1] ? 0 : 1));
  return d[m][n];
}
function nearest(name, pool) {
  const segsA = new Set(name.split("_"));
  let best = null, bestScore = -1;
  for (const c of pool) {
    const segsB = new Set(c.split("_"));
    let inter = 0; for (const s of segsA) if (segsB.has(s)) inter++;
    const jac = inter / new Set([...segsA, ...segsB]).size;
    const lev = 1 - levenshtein(name, c) / Math.max(name.length, c.length);
    const score = jac * 0.7 + lev * 0.3;
    if (score > bestScore) { bestScore = score; best = c; }
  }
  return { candidate: best, score: bestScore };
}

// ---------------------------------------------------------------- run
const runtime = RUNTIME_JSON && existsSync(RUNTIME_JSON)
  ? Object.fromEntries(JSON.parse(readFileSync(RUNTIME_JSON, "utf8"))
      .filter((r) => r.ok).map((r) => [r.vendor, r.tools]))
  : {};

// Third source: the gateway's pinned prod schemas (schemas/<vendor>.json), captured
// from the image digests in .harness/vendors.json. Available for 12 vendors only.
function schemaTools(vendorId) {
  if (!vendorId) return null;
  const p = join(GATEWAY, "schemas", `${vendorId}.json`);
  if (!existsSync(p)) return null;
  try {
    const j = JSON.parse(readFileSync(p, "utf8"));
    return (j.tools || []).map((t) => t.name).sort();
  } catch { return null; }
}

const live = LIVE_JSON && existsSync(LIVE_JSON) ? JSON.parse(readFileSync(LIVE_JSON, "utf8")) : {};

const report = [];
for (const dir of findPluginDirs(PLUGINS)) {
  const slug = relative(PLUGINS, dir);
  if (SKIP.has(slug)) continue;
  const mapped = PLUGIN_MAP[slug] ?? { server: `${slug.split("/").pop()}-mcp`, vendor: slug.split("/").pop() };
  const serverDir = mapped.server;
  const st = serverDir ? staticTools(serverDir) : null;
  const rt = serverDir ? runtime[serverDir] : null;
  const sc = mapped.hostedNote ? null : schemaTools(mapped.vendor);
  const lv = mapped.vendor && live[mapped.vendor] ? live[mapped.vendor] : null;
  const truth = st || rt || sc || lv
    ? [...new Set([...(st || []), ...(rt || []), ...(sc || []), ...(lv || [])])].sort()
    : null;

  const docs = documentedNames(dir);
  const entry = {
    plugin: slug, server: serverDir, vendor: mapped.vendor,
    hostedNote: mapped.hostedNote || null,
    groundTruth: truth,
    groundTruthSource: truth
      ? [st && "static", rt && "runtime", sc && "prod-schema", lv && "live-gw"].filter(Boolean).join("+")
      : "NONE",
    runtimeCount: rt ? rt.length : null, staticCount: st ? st.length : null,
    schemaCount: sc ? sc.length : null, liveCount: lv ? lv.length : null,
    documented: [], drifted: [], undocumented: [], unverified: !truth,
    // Self-check: anything the live server advertised that the static scan missed.
    staticMissed: st && rt ? rt.filter((t) => !st.includes(t)) : [],
  };
  if (!truth) {
    // No ground truth (hosted upstream / unbuilt server). Still record what the
    // docs claim, using the plugin slug as the namespace guess, so the vendor
    // lands in the "unverified" bucket with its claim surface visible.
    const guess = slug.split("/").pop().replace(/-/g, "_");
    const alt = slug.split("/").pop().split("-")[0];
    entry.documentedUnverified = [...docs.keys()].filter((n) => {
      const head = n.split("_")[0];
      return n.startsWith(guess + "_") || head === alt;
    }).sort();
    entry.allDocTokens = [...docs.keys()].sort();
    report.push(entry); continue;
  }

  const prefixes = prefixesOf(truth);
  const truthSet = new Set(truth);
  for (const [name, sites] of docs) {
    if (!prefixes.has(name.split("_")[0])) continue; // not a tool of this server's namespace
    entry.documented.push(name);
    if (truthSet.has(name)) continue;
    const { candidate, score } = nearest(name, truth);
    entry.drifted.push({
      name, sites: sites.slice(0, 6), occurrences: sites.length,
      classification: score >= 0.45 ? "renamed" : "absent",
      realName: score >= 0.45 ? candidate : null, similarity: +score.toFixed(2),
    });
  }
  const docSet = new Set(entry.documented);
  entry.undocumented = truth.filter((t) => !docSet.has(t));
  entry.documented.sort(); entry.drifted.sort((a, b) => a.name.localeCompare(b.name));
  report.push(entry);
}

if (JSON_OUT) writeFileSync(JSON_OUT, JSON.stringify(report, null, 2));

// ---------------------------------------------------------------- console summary
const drifted = report.filter((r) => r.drifted.length);
const clean = report.filter((r) => !r.unverified && !r.drifted.length);
const unver = report.filter((r) => r.unverified);
console.log("PLUGIN".padEnd(38) + "TRUTH".padEnd(7) + "DOC'D".padEnd(7) + "DRIFT".padEnd(7) + "UNDOC".padEnd(7) + "SOURCE");
for (const r of report) {
  console.log(
    r.plugin.padEnd(38) +
    String(r.groundTruth ? r.groundTruth.length : "-").padEnd(7) +
    String(r.documented.length).padEnd(7) +
    String(r.drifted.length).padEnd(7) +
    String(r.undocumented.length).padEnd(7) +
    r.groundTruthSource,
  );
}
console.log(`\ndrifted=${drifted.length} clean=${clean.length} unverified=${unver.length} total=${report.length}`);
process.exitCode = drifted.length ? 1 : 0;

probe-tools.mjs (runtime source)

#!/usr/bin/env node
// Probe local *-mcp servers over stdio for their real tools/list surface.
import { spawn } from "node:child_process";
import { existsSync, readdirSync, writeFileSync, mkdirSync } from "node:fs";
import { join } from "node:path";

const ROOT = process.argv[2] || "/Users/asachs/mcp";
const OUT = process.argv[3];

const dirs = readdirSync(ROOT, { withFileTypes: true })
  .filter((d) => d.isDirectory() && d.name.endsWith("-mcp"))
  .map((d) => d.name)
  .sort();

// Generic fake credentials — enough to get past constructor-time validation in
// most servers. Tool listing itself never hits the vendor API.
const FAKE = "probe-dummy-value";
function fakeEnv(vendor) {
  const base = {
    ...process.env,
    MCP_TRANSPORT: "stdio",
    TRANSPORT: "stdio",
    NODE_ENV: "test",
  };
  // Blanket-fill anything that looks like a credential var name the server may read.
  const suffixes = [
    "API_KEY", "APIKEY", "API_TOKEN", "TOKEN", "SECRET", "CLIENT_ID",
    "CLIENT_SECRET", "USERNAME", "USER", "PASSWORD", "ACCOUNT_ID", "ORG_ID",
    "ORGANIZATION_ID", "TENANT_ID", "SUBDOMAIN", "DOMAIN", "BASE_URL", "URL",
    "REGION", "INTEGRATION_CODE", "PARTNER_ID", "COMPANY_ID", "SITE_ID",
    "INSTANCE", "HOST", "EMAIL", "KEY", "ID",
  ];
  const prefix = vendor.replace(/-mcp$/, "").replace(/-/g, "_").toUpperCase();
  for (const s of suffixes) base[`${prefix}_${s}`] = FAKE;
  base[prefix] = FAKE;
  return base;
}

function probe(vendorDir) {
  return new Promise((resolve) => {
    const cwd = join(ROOT, vendorDir);
    const entry = join(cwd, "dist", "index.js");
    if (!existsSync(entry)) {
      return resolve({ vendor: vendorDir, ok: false, reason: "no dist/index.js" });
    }
    const child = spawn("node", [entry], {
      cwd,
      env: fakeEnv(vendorDir),
      stdio: ["pipe", "pipe", "pipe"],
    });
    let buf = "";
    let stderr = "";
    let done = false;
    const finish = (res) => {
      if (done) return;
      done = true;
      try { child.kill("SIGKILL"); } catch { /* noop */ }
      resolve(res);
    };
    const timer = setTimeout(
      () => finish({ vendor: vendorDir, ok: false, reason: "timeout", stderr: stderr.slice(-500) }),
      20000,
    );

    const send = (obj) => {
      try { child.stdin.write(JSON.stringify(obj) + "\n"); } catch { /* noop */ }
    };

    child.stdout.on("data", (d) => {
      buf += d.toString();
      let idx;
      while ((idx = buf.indexOf("\n")) >= 0) {
        const line = buf.slice(0, idx).trim();
        buf = buf.slice(idx + 1);
        if (!line.startsWith("{")) continue;
        let msg;
        try { msg = JSON.parse(line); } catch { continue; }
        if (msg.id === 1) {
          send({ jsonrpc: "2.0", method: "notifications/initialized" });
          send({ jsonrpc: "2.0", id: 2, method: "tools/list", params: {} });
        } else if (msg.id === 2) {
          clearTimeout(timer);
          if (msg.result?.tools) {
            finish({
              vendor: vendorDir,
              ok: true,
              tools: msg.result.tools.map((t) => t.name).sort(),
              descriptions: Object.fromEntries(
                msg.result.tools.map((t) => [t.name, (t.description || "").slice(0, 300)]),
              ),
            });
          } else {
            finish({ vendor: vendorDir, ok: false, reason: "no tools in result", raw: JSON.stringify(msg).slice(0, 400) });
          }
        }
      }
    });
    child.stderr.on("data", (d) => { stderr += d.toString(); });
    child.on("error", (e) => finish({ vendor: vendorDir, ok: false, reason: "spawn error: " + e.message }));
    child.on("exit", (code) => {
      if (!done) {
        clearTimeout(timer);
        finish({ vendor: vendorDir, ok: false, reason: `exited ${code}`, stderr: stderr.slice(-600) });
      }
    });

    send({
      jsonrpc: "2.0",
      id: 1,
      method: "initialize",
      params: {
        protocolVersion: "2025-06-18",
        capabilities: {},
        clientInfo: { name: "drift-audit-probe", version: "1.0.0" },
      },
    });
  });
}

const results = [];
const CONCURRENCY = 6;
let cursor = 0;
async function worker() {
  while (cursor < dirs.length) {
    const v = dirs[cursor++];
    const r = await probe(v);
    results.push(r);
    process.stderr.write(`${r.ok ? "OK  " : "FAIL"} ${v}${r.ok ? ` (${r.tools.length})` : ` — ${r.reason}`}\n`);
  }
}
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
results.sort((a, b) => a.vendor.localeCompare(b.vendor));
const json = JSON.stringify(results, null, 2);
if (OUT) { mkdirSync(join(OUT, ".."), { recursive: true }); writeFileSync(OUT, json); }
else console.log(json);

[asachs01]

Script PR is up: #180 (tooling only — adds scripts/tool-drift-audit.mjs and scripts/probe-mcp-tools.mjs, touches no plugin content). Deliberately not wired into validate.yml yet; the CI-wiring decision is a follow-up described there.

[asachs01]

Re-ran the audit against main @ 792c422 (after #179 and #181 landed). Drift sets are byte-identical to the 44c57a4 run — 17 script-detected plugins, same names, same sites; 34 clean, 13 script-unverified. Neither the reciprocal-anti-trigger pass nor the dangling-skill-reference repair touched a tool name, so every finding above still stands as written.