🚨 On-Call Specialist Agent

AI-powered on-call assistant for Microsoft ICM — from incident triage to R2D generation

Features • Quick Start • Skills • Architecture • Commands

---

✨ Features

Feature	Description
🔍 Incident Investigation	Query ICM incidents, get AI summaries, find similar incidents, check customer impact
📋 R2D Generation	Auto-generate Ready-to-Dispatch work items in Azure DevOps from ICM incidents
🔬 Kusto Investigation	3-phase Kusto pipeline: ApiQoSEvent → ContextActivity → DisksBI
🌐 ICM Crawler	Puppeteer-based crawler for authored summaries & discussion entries (data the API doesn't expose)
🛠️ 23+ ICM MCP Tools	Full access to incidents, teams, on-call schedules, customer impact via MCP protocol
🧩 Extensible Skills	Add new incident types by creating a skill markdown file — no code changes needed
📊 ADO Integration	Create/update Azure DevOps work items via REST API with HTML formatting
👥 Team & Contact Lookup	Find on-call engineers, search teams, lookup contacts by alias

---

📋 Prerequisites

• ✅ Node.js v18+ (Download)
• ✅ Azure CLI installed and logged in (Install Guide)
• ✅ Microsoft Edge (for ICM Crawler and Kusto Web queries)
• ✅ ICM Access via your Microsoft account
• ✅ VPN connected (for Kusto cluster access)

---

🚀 Quick Start

1. Clone & Install

git clone https://github.com/zhanrz88-sudo/oncallspecialist.git
cd oncallspecialist
npm run setup

2. Authenticate

# Login to Azure for ICM API access
az login --scope api://icmmcpapi-prod/mcp.tools

# Login to Azure for ADO access (if using R2D generation)
az login

3. First-Time Setup (ICM Crawler & Kusto)

The ICM crawler and Kusto queries use Edge with a persistent profile. On first use, a browser window opens for interactive login. After that, everything runs headlessly.

4. Use It

# Direct CLI
node oncall-agent/oncall-agent.js incident 123456789

# Or install for GitHub Copilot CLI
npm run install-copilot

---

🧩 Skills

Skills are markdown-defined workflows that the agent executes step-by-step. Each skill is a self-contained playbook for a specific on-call task.

Skill	Trigger	What It Does
Triage Incident	"triage incident <id>"	Parallel investigation: incident details, impact, similar incidents, mitigation hints
Auto Mitigate	"mitigate incident <id>"	Match incident to known patterns, suggest fix, provide TSG links
Generate R2D	"generate R2D for <id>"	Full pipeline: ICM → Crawler → Kusto → R2D template → ADO work item
DES Weak Association	"break DES association"	Specialized R2D for DES deletion blocked by stale storage account references

Adding a New Skill

1. Create skills/skill-r2d-<name>.md following the existing template format
2. Define the 8 R2D questions with pre-filled answers for that incident type
3. Register it in skills/skill-generate-r2d.md → Sub-Skills Registry table
4. The agent will auto-match it based on incident patterns

---

🏗️ Architecture

oncall-specialist/
├── oncall-agent/           # CLI agent + ICM client library
│   ├── oncall-agent.js     # Main CLI (incident, oncall, kusto, etc.)
│   ├── icm-client.js       # Reusable ICM API client
│   └── *.md                # Agent instructions + skill files
├── icm-mcp-server/         # MCP server for GitHub Copilot integration
│   ├── index.js            # MCP protocol server (23+ tools)
│   ├── icm-crawler.js      # Puppeteer-based ICM portal crawler
│   ├── kusto-web-query.js  # Kusto Web Explorer query runner
│   └── kusto-query.js      # Kusto SDK query runner
├── skills/                 # Skill definitions (shareable)
│   ├── skill-generate-r2d.md
│   ├── skill-r2d-break-des-storage-association.md
│   ├── skill-triage-incident.md
│   └── skill-auto-mitigate.md
├── docs/                   # TSG documents
│   ├── tsg-refresh-encryption-scopes.md
│   ├── tsg-constructing-blob-uri.md
│   └── ...
└── package.json            # Root package with setup scripts

Data Flow: R2D Generation

ICM Incident ID
    │
    ├──► ICM API (23 MCP tools)  ──► title, severity, subscription, region
    ├──► ICM Crawler (Puppeteer) ──► authored summary, discussion entries
    │
    ▼
Kusto 3-Phase Pipeline
    ├──► Phase 1: DiskManagerApiQoSEvent      ──► failed DELETEs, correlation IDs
    ├──► Phase 2: DiskManagerContextActivity   ──► root cause, blocking storage account
    └──► Phase 3: DisksBI/StorageAccount       ──► validate DES ↔ storage association
    │
    ▼
Sub-Skill Match (pattern-based)
    ├──► DES weak association template
    ├──► Snapshot stale reference template
    └──► (extensible — add your own)
    │
    ▼
R2D Template Engine
    ├──► 8-question HTML description
    ├──► Iteration path auto-calculation
    └──► Acceptance criteria
    │
    ▼
Azure DevOps REST API
    └──► Create/Update PBI with tags, area path, assignment

---

📖 Commands

On-Call Agent CLI

# Incident operations
node oncall-agent/oncall-agent.js incident <id>        # Get incident details + AI summary
node oncall-agent/oncall-agent.js impact <id>           # Customer impact
node oncall-agent/oncall-agent.js find-similar <id>     # Smart similar incident search
node oncall-agent/oncall-agent.js mitigate <id>         # Mitigation hints

# Team & contact
node oncall-agent/oncall-agent.js oncall "Disk Service" # Who is on-call
node oncall-agent/oncall-agent.js team "Disk Service"   # Search teams
node oncall-agent/oncall-agent.js contact <alias>       # Lookup person

# Kusto queries
node oncall-agent/oncall-agent.js kusto disks Disks "<KQL>"
node oncall-agent/oncall-agent.js kusto disksbi DisksBI "<KQL>"
node oncall-agent/oncall-agent.js kusto-tables disks Disks
node oncall-agent/oncall-agent.js kusto-schema disks Disks <table>

ICM Crawler

# Full crawl (authored summary + discussion entries)
node icm-mcp-server/icm-crawler.js <incidentId>

# Summary only
node icm-mcp-server/icm-crawler.js <incidentId> --summary-only

Kusto Web Query

node icm-mcp-server/kusto-web-query.js https://disks.kusto.windows.net Disks "<KQL query>"

---

🔧 Configuration

Custom Instructions for GitHub Copilot CLI

Add to your Copilot custom instructions to enable the agent:

You have access to an On-Call Specialist Agent.
Run commands via: node <path>/oncall-agent/oncall-agent.js <command> [args]

See oncall-agent/oncall-specialist.instructions.md for the full instruction template.

Key Team Reference

Team	ICM Public ID	Team ID
Disk Service	AZURERT\DiskService	28322

Kusto Clusters

Alias	URL	Database
disks	https://disks.kusto.windows.net	Disks
disksbi	https://disksbi.kusto.windows.net	DisksBI

---

🤝 Contributing

1. New Skills: Create a skill-*.md in skills/ following the existing format
2. New TSGs: Add tsg-*.md in docs/
3. New Commands: Extend oncall-agent/oncall-agent.js
4. Bug Fixes: PRs welcome!

---

📄 License

MIT License — see LICENSE for details.

---

Made with ❤️ for the Microsoft on-call engineering community