feat: add debugging-with-ably-cli implicit skill

[mattheworiordan]

Why a Separate Skill?

The existing using-ably skill teaches agents how to build with Ably — product selection, SDK patterns, authentication, React integration. But when things break during development, agents need a different kind of knowledge: how to observe, simulate, and diagnose.

These are distinct cognitive modes. Building requires architectural judgment ("which SDK?", "JWT or token auth?"). Debugging requires investigative reasoning ("what's actually happening on this channel?", "can I simulate the other end?"). Combining them into one skill would create the kind of bloated, procedural content that SkillsBench research shows degrades agent performance (-2.9pp for comprehensive procedural skills).

The Problem This Solves

We watched an agent try to debug a presence discovery issue between a JSON-RPC client and server. Without knowledge of the Ably CLI, the agent:

1. Used Chrome DevTools to take screenshots and read console output
2. Injected console.log statements into React components
3. Tried (and failed) to use evaluate_script via Chrome MCP
4. Reloaded pages repeatedly, checking console output each time
5. Eventually identified the symptom ("createRpcSession hanging") but couldn't determine why presence wasn't working

This took 20+ tool calls across two browser tabs and still didn't reach root cause. With the CLI, the agent could have run:

# See who's actually on the presence channel
ably channels presence subscribe rpc:jsonrpc:presence

# Check if the channel even exists
ably channels list --prefix rpc:jsonrpc

# Watch live messages flowing (or not)
ably channels subscribe "rpc:jsonrpc:*"

# Simulate the server end to test the client
ably channels presence enter rpc:jsonrpc:presence --client-id test-server --data '{"role":"server"}'

Any single one of these would have immediately revealed whether the issue was presence, channel naming, auth, or transport — in one command instead of 20+ DevTools round-trips.

Design Philosophy

The skill is designed around three principles from SkillsBench research:

1. Compact domain knowledge, not procedures (+18.8pp improvement). The skill provides decision trees, state reference tables, and debugging facts — not step-by-step workflows.

2. Capabilities, not commands. The skill teaches what the CLI can do (observe, simulate, manage) and when to use it (diagnostic decision tree), not every flag and argument. Agents discover exact syntax via ably --help at runtime. This means the skill won't go stale as the CLI evolves.

3. Front-load the highest-value content. Section 1 is the diagnostic decision tree (the entry point to every debugging session). Section 2 is key debugging facts (domain knowledge agents genuinely lack). Auth and capability descriptions come after.

What's Included

Section	Purpose
Diagnostic Decision Tree	Symptom → approach mapping for messages, presence, connections, auth, channels, Chat, Spaces
Key Debugging Facts	Non-obvious domain knowledge: persistence defaults, capability scoping, presence requirements, case sensitivity
CLI Authentication	How to get credentials (API key from .env or ably login)
What the CLI Can Do	Capability categories: Observe, Simulate, Manage, Get Help
Long-Running Commands	Background task pattern for subscribe/presence/stats commands
State Reference	Connection and channel state machines with diagnostics
Environment Gotchas	SSR, serverless, StrictMode, CSP, proxy, mobile

Plus an AGENTS.md maintenance guide explaining the design philosophy and when to update vs. when not to.

189 lines / ~2,000 tokens — well under the 500-line budget.

What's not included

Given we have no way of publishing a stream of tokens or subscribing to mutual messages now, CLI clearly does not attempt to do that, but I think it absolutely should, especially in the context of AIT.

I believe there's someone working on this right now, but as soon as that's available, the skills will be updated.

It also has no capabilities of live objects, but that's because the CLI does not support that, and I think we should really just go ahead and add some very basic live object support regardless, so that this CLI has richer capabilities to debug the things that are going to be needed for AIT specifically, which is where the focus is.

Test plan

• Verify skill triggers when an agent encounters Ably debugging scenarios
• Verify skill does NOT trigger for build/integration tasks (using-ably handles those)
• Verify decision tree covers all Ably products the CLI supports
• Verify ably --help matches the capability categories described
• Review AGENTS.md maintenance guide for clarity

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds a new debugging-with-ably-cli implicit skill that teaches AI agents how to use the Ably CLI to diagnose Ably-specific issues (messages not arriving, presence failures, auth errors, connection problems). It is intentionally scoped to debugging to complement (not duplicate) the existing using-ably skill, which covers building new integrations. The skill is designed around SkillsBench research principles: compact domain knowledge over procedures, capability-oriented rather than command-reference, and front-loaded high-value content.

Changes:

• Adds debugging-with-ably-cli/SKILL.md: the skill itself, covering a diagnostic decision tree, key debugging facts, CLI auth, capability categories, long-running command guidance, connection/channel state reference tables, and environment-specific gotchas.
• Adds debugging-with-ably-cli/AGENTS.md: a maintenance guide explaining the design philosophy, update triggers, and validation checklist.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
debugging-with-ably-cli/SKILL.md	New skill file with diagnostic decision tree, debugging facts, CLI capability descriptions, state reference tables, and environment gotchas
debugging-with-ably-cli/AGENTS.md	New maintenance guide documenting when and how to update the skill as the Ably CLI evolves

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mattheworiordan]

Just ran it to test a real debugging issue for a client. Pretty successful 👍

Please use the debugging-with-ably-cli skill and start debugging the underlying issue:

⏺ Good idea — let me use the Ably CLI to observe the presence channel live and see what's actually happening. First I need the API key.

⏺ Searched for 1 pattern, read 2 files (ctrl+o to expand)

⏺ Now let me use the Ably CLI to subscribe to presence on the presence:jsonrpc:lobby channel in the background, then open the client and server in the browser to see what presence events actually flow.

⏺ Bash(ably channels presence subscribe "presence:jsonrpc:lobby" --api-key "[[OBFUSCATED]]" --duration 30 2>&1 &
      echo "Prese…)
  ⎿  Running in the background (↓ to manage)

⏺ Good. Now let me subscribe to presence and also check who's currently present on the channel.

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels presence subscribe "presence:jsonrpc:lobby" --api-key "[[OBFUSCATED]]" --du…)
  ⎿  Running in the background (↓ to manage)

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels occupancy get "presence:jsonrpc:lobby" --api-key "[[OBFUSCATED]]" 2>&1)
  ⎿  Occupancy metrics for channel 'presence:jsonrpc:lobby':

     Connections: 2
     … +5 lines (ctrl+o to expand)

⏺ There's still 1 stale presence member and 2 connections from previous tests. Let me close all browser pages first, wait for cleanup, then do a clean test.

⏺ Bash(cd /Users/matthew.oriordan/.claude-work/plugins/cache/dev-browser-marketplace/dev-browser/66ef8b3c3f93/skills/dev-browser && npx tsx <<'EOF'
      import { connectLit…)
  ⎿  Open pages: []
  ⎿  Shell cwd was reset to /Users/matthew.oriordan/Projects/Ably/demos/capnprotorpc

⏺ Bash(sleep 15 && ASDF_NODEJS_VERSION=24.11.0 ably channels occupancy get "presence:jsonrpc:lobby" --api-key "s57drg.3bnE1Q:FgDKG78ZG-G0afC2OzSM2ms8QJRNMnHGHnTmR_sRhQ…)
  ⎿  Occupancy metrics for channel 'presence:jsonrpc:lobby':

     Connections: 2
     … +5 lines (ctrl+o to expand)
  ⎿  (timeout 25s)

⏺ Still stale connections. Let me check who's actually present.

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels presence --help 2>&1 | head -20)
  ⎿  USAGE
       $ ably channels presence
     … +6 lines (ctrl+o to expand)

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels list --prefix "presence:jsonrpc" --api-key "[[OBFUSCATED]]" 2>&1)
  ⎿  Found 1 active channels:
     presence:jsonrpc:lobby
       Connections: 2
     … +4 lines (ctrl+o to expand)

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels history "presence:jsonrpc:lobby" --api-key "[[OBFUSCATED]]" 2>&1)
  ⎿  No messages found in the channel history.

⏺ Background command "Subscribe to presence events on jsonrpc lobby for 45s" completed (exit code 0)

⏺ Read 1 file (ctrl+o to expand)

⏺ There's a stale client client-tv1mh8 still present. Let me start a fresh presence subscribe, open fresh pages, and observe the actual enter events.

⏺ Bash(ASDF_NODEJS_VERSION=24.11.0 ably channels presence subscribe "presence:jsonrpc:lobby" --api-key "[[OBFUSCATED]]" --du…)
  ⎿  Running in the background (↓ to manage)

⏺ Bash(cd /Users/matthew.oriordan/.claude-work/plugins/cache/dev-browser-marketplace/dev-browser/66ef8b3c3f93/skills/dev-browser && npx tsx <<'EOF'
      import { connectLit…)
  ⎿  Client opened, waiting 5s...
     Server opened, waiting 8s...
     Client state: {"url":"http://localhost:5173/?protocol=jsonrpc&role=client","hasServer":false,"noServer":true,"snippet":"←\nClient\nclient-t4uyff\nJSON-RPC 2.0\n\nRemote counter controls\n\nNo server\nConnected\nWai
     ting for server...\n\nOpen a server tab to start the counter API.\n\n"}
  ⎿  Shell cwd was reset to /Users/matthew.oriordan/Projects/Ably/demos/capnprotorpc
  ⎿  (timeout 30s)

⏺ Still failing. Let me check the presence events the CLI captured.

⏺ Task Output(non-blocking) b7nijp4ol
  ⎿  Subscribing to presence events on channel: presence:jsonrpc:lobby
     Listening for presence events. Press Ctrl+C to exit.
     [2026-03-04T17:13:03.482Z] Channel: presence:jsonrpc:lobby | Action: present | Client: client-tv1mh8
     … +13 lines (ctrl+o to expand)

⏺ The CLI confirms the server's enter event IS being delivered on the Ably side (at 17:18:20). So the event reaches the client's Ably connection, but the handler isn't updating the UI. Let me add targeted console.log
  debugging to trace the issue.

[mattheworiordan]

@umair-ably I see you added 10a6c1c. Funnily enough I had an almost identical commit as I was waiting for #2 to land. When I tried to push I saw the conflict. I've added a few small changes on top f3153c5.

I'd like to merge this as I strongly believe if we dog food these skills ourselves, we'll get these really good quickly too but constantly updating as we find issues. I'll ping the others as no one has approved/requested changes/declined this PR. I want to merge and improved.

[paddybyers]

LGTM, but I added some suggestions

[umair-ably]

@mattheworiordan skill looks good!

We've slightly deviated from where the CLI is right now in the midst of the GA efforts... I've addressed this in 0d12d5b

but we need to release a new version of the cli first... can you approve this please?

[mattheworiordan]

@umair-ably would you mind picking up the feedback from @paddybyers and the bot and getting this PR over the line i.e. are you OK for me to just leave it with you now?

On the queues commenbt, I think we should extend the skill to explain how queues work and debugging instead of removing though.

[paddybyers]

LGTM, thanks