Copilot Compass

Navigate your GitHub Copilot adoption with actionable insights

An MCP App that delivers comprehensive GitHub Copilot usage analytics through an interactive React dashboard, powered by the MCP Apps SDK.

Features • Quick Start • Codespaces • Usage • Architecture • Troubleshooting

Overview

Copilot Compass is a Model Context Protocol (MCP) application that transforms raw GitHub Copilot metrics into actionable intelligence. Built with the MCP Apps SDK, it provides both AI-readable data and an interactive visual dashboard that renders directly in MCP-compatible hosts like VS Code and Claude Desktop.

What Makes This Special?

Unlike traditional dashboards, Copilot Compass leverages the MCP Apps SDK pattern:

AI-First: Tools return structured data that AI models can analyze and summarize
Visual Dashboard: The same tool invocation renders an interactive React UI in the host
Real-Time Sync: UI automatically updates when new data arrives via ontoolresult
Host Theming: Dashboard adapts to the host's color scheme via useHostStyles

Features

Analytics Dashboard

Active Users Trend — 14-day line chart with daily active user counts
Acceptance Rate — Track code suggestion acceptance over time
Language Breakdown — Top languages by suggestions, acceptances, and engaged users
Editor Distribution — Usage split across VS Code, JetBrains, Neovim, and more
Chat Metrics — Sessions, code insertions, and copy events from Copilot Chat
PR Intelligence — Pull request summary generation statistics

Understanding the Metrics

Metric	What It Measures	Why It Matters
Active Users	Unique users who received Copilot suggestions	Adoption breadth
Engaged Users	Users who accepted at least one suggestion	Active value extraction
Acceptance Rate	Accepted suggestions ÷ total suggestions	Quality of suggestions
Lines Accepted	Actual code lines integrated from Copilot	Productivity impact
Chat Sessions	Copilot Chat conversations initiated	AI pair programming usage
Code Insertions	Chat-generated code added to files	Conversational coding value
PR Summaries	Pull request descriptions auto-generated	Review process efficiency

Report Generation

Enterprise & Organization Support — Fetch metrics at enterprise or org level
Flexible Date Ranges — Analyze any time period
Automatic Fallback — Demo mode with realistic mock data when API unavailable
Caching Layer — 5-minute TTL reduces API calls and improves response times

MCP Apps SDK Features

Tool + UI Binding — _meta.ui.resourceUri links tools to their visual representation
App-Only Tools — visibility: ["app"] for UI-initiated actions hidden from AI
Host Context — Safe area insets and theme integration
Lifecycle Hooks — ontoolresult, ontoolinput, onteardown callbacks

Quick Start

Prerequisites

Node.js 18+
VS Code Insiders — Required for MCP Apps SDK (interactive dashboard). Download here. VS Code Stable supports MCP tools but not the visual dashboard UI yet.
GitHub PAT with scopes:
- manage_billing:copilot — Access Copilot metrics
- read:enterprise — Enterprise-level access
- read:org — Organization metrics

Installation

# Clone the repository
git clone https://github.com/samueltauil/copilot-compass.git
cd copilot-compass

# Install dependencies
npm install

# Configure environment
cp .env.example .env
# Edit .env and add your GITHUB_TOKEN

Build & Run

# Build the React dashboard and TypeScript server
npm run build

# Start the MCP server
npm start

The server starts at http://localhost:3001 with:

MCP Endpoint: http://localhost:3001/mcp
Health Check: http://localhost:3001/health

Connect Your MCP Client

Add to your MCP client configuration (e.g., Claude Desktop, VS Code):

For remote HTTP connection:

{
  "mcpServers": {
    "copilot-compass": {
      "type": "http",
      "url": "http://localhost:3001/mcp"
    }
  }
}

For local stdio connection (alternative):

{
  "mcpServers": {
    "copilot-compass": {
      "type": "stdio",
      "command": "node",
      "args": ["dist/server.js"],
      "cwd": "/path/to/copilot-compass",
      "env": {
        "GITHUB_TOKEN": "your-github-token"
      }
    }
  }
}

For VS Code users:

Copy the included example file to create your MCP config:

cp .vscode/mcp.json.example .vscode/mcp.json

The default uses localhost:3001. If using a tunnel (cloudflared, ngrok), edit .vscode/mcp.json to use your tunnel URL:

{
  "servers": {
    "copilot-compass": {
      "type": "http",
      "url": "https://your-tunnel-url.trycloudflare.com/mcp"
    }
  }
}

Note: .vscode/mcp.json is gitignored so your local config won't be overwritten.

Run in GitHub Codespaces

⚠️ Important: MCP Apps SDK (interactive dashboard UI) currently requires VS Code Insiders. The Codespaces web editor uses stable VS Code. To use the full dashboard UI, you must connect from VS Code Insiders on your local machine.

📌 Note: MCP Apps SDK support is expected to ship in VS Code Stable in a future release. See the official MCP Apps documentation and the MCP Apps SDK repository for the latest updates.

Option A: Connect from VS Code Insiders Desktop (Full UI)

This gives you the complete MCP Apps experience with the interactive dashboard.

Install VS Code Insiders locally
Install the GitHub Codespaces extension in VS Code Insiders
Create a Codespace at github.com/samueltauil/copilot-compass
In the Codespace web view, click the hamburger menu (≡) → Open in VS Code Desktop
When prompted, choose Visual Studio Code Insiders

Option B: Use Codespaces Web Editor (Tools Only)

If you just need the MCP tools (not the visual dashboard), the web editor works fine:

Or manually:

Go to github.com/samueltauil/copilot-compass
Click Code → Codespaces → Create codespace on main
Wait for the container to build (~2 minutes)

What Happens Automatically

When your Codespace starts:

✅ Dependencies are installed and project is built
✅ MCP server starts automatically in the background on port 3001
✅ Port 3001 is made public with HTTPS
✅ MCP config is created at /home/codespace/.vscode-remote/data/Machine/mcp.json
✅ MCP settings are enabled (chat.mcp.enabled, chat.mcp.discovery.enabled)

The MCP server URL is: http://localhost:3001/mcp (via port forwarding)

Configure GitHub Token (Optional)

For live Copilot metrics (instead of demo data), add your GitHub token as a Codespaces secret:

Go to github.com/settings/codespaces
Click New secret
Name: GITHUB_TOKEN
Value: Your PAT with manage_billing:copilot, read:enterprise, read:org scopes
Repository access: Select the repository where you're running the Codespace
Apply the secret to your Codespace:
- If Codespace is not running yet: Just create it — the secret will be available automatically
- If Codespace is already running: Go to github.com/codespaces, click the ... menu on your Codespace → Stop, then Start again
- Quick restart from terminal: Press Ctrl+Shift+P → "Codespaces: Stop Current Codespace", then reopen

Alternatively, create a .env file in the Codespace terminal:

echo "GITHUB_TOKEN=your-token-here" > .env
npm start  # Restart the server

Verify It's Working

The MCP server starts automatically in the background. Check the logs:

cat /tmp/mcp-server.log

You should see:

🧭 Copilot Compass - Codespaces Setup
✅ Codespaces detected: your-codespace-name
✅ MCP config written to: ~/.vscode-server/data/User/mcp.json
🚀 Starting MCP server...
✅ Port 3001 is now public

Open the Ports tab and verify port 3001 shows Public visibility
- If it shows Private, right-click the port → Port Visibility → Public
- Or run: gh codespace ports visibility 3001:public -c $CODESPACE_NAME
The MCP server is now ready! Use it from Copilot Chat — MCP discovery is enabled automatically.

Manual Start (If Needed)

If the server stops or you need to restart:

# Quick restart
npm start

# Full setup with config
nohup bash .devcontainer/setup.sh > /tmp/mcp-server.log 2>&1 &

Tip: The Codespaces URL persists as long as your Codespace is running. For long-term use, consider keeping the Codespace alive or using a dedicated VM.

Public Access via Tunnel (Local Development)

To expose your local MCP server to the internet (useful for sharing, mobile access, or cloud-based MCP clients), you can use a tunneling service.

Option 1: Cloudflare Tunnel (Recommended)

Cloudflared provides free, secure tunnels with automatic HTTPS.

# Install cloudflared (Windows)
winget install cloudflare.cloudflared

# Install cloudflared (macOS)
brew install cloudflared

# Install cloudflared (Linux)
curl -L https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -o cloudflared
chmod +x cloudflared

Start the tunnel:

# In one terminal, start the MCP server
npm start

# In another terminal, create the tunnel
cloudflared tunnel --url http://localhost:3001

Cloudflared outputs a public URL like https://random-words.trycloudflare.com. Use this in your MCP client config:

{
  "mcpServers": {
    "copilot-compass": {
      "type": "http",
      "url": "https://random-words.trycloudflare.com/mcp"
    }
  }
}

Note: The free trycloudflare.com URL changes each time you restart the tunnel. For a persistent URL, set up a named tunnel with your own domain.

Option 2: ngrok

ngrok is another popular tunneling option with a free tier.

# Install ngrok
npm install -g ngrok

# Or download from https://ngrok.com/download

Start the tunnel:

# Start the MCP server
npm start

# In another terminal
ngrok http 3001

ngrok provides a URL like https://abc123.ngrok.io. Update your MCP config accordingly.

Option 3: Tailscale Funnel

If you use Tailscale, you can use Funnel to expose services.

tailscale funnel 3001

Running as a Background Service

For production or persistent deployments:

Using PM2 (Node.js process manager):

# Install PM2
npm install -g pm2

# Start the server
pm2 start dist/server.js --name copilot-compass

# View logs
pm2 logs copilot-compass

# Stop the server
pm2 stop copilot-compass

# Auto-start on system boot
pm2 startup
pm2 save

Using systemd (Linux):

Create /etc/systemd/system/copilot-compass.service:

[Unit]
Description=Copilot Compass MCP Server
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/copilot-compass
ExecStart=/usr/bin/node dist/server.js
Restart=on-failure
Environment=GITHUB_TOKEN=your-token
Environment=PORT=3001

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable copilot-compass
sudo systemctl start copilot-compass

Architecture

flowchart TB
    subgraph Host["MCP Host (VS Code, Claude Desktop)"]
        User([User]) --> Model[AI Model]
        Model -->|Tool Call| Server
        Server -->|JSON Report| Model
        Model -->|Summary| Response([Response])
        
        subgraph Dashboard["React Dashboard"]
            useApp["useApp() hook"]
            Charts["Chart.js Visualizations"]
        end
    end
    
    subgraph Server["Copilot Compass Server"]
        Tools["registerAppTool()
        • generate_copilot_report
        • refresh_report"]
        Resources["registerAppResource()
        • ui://copilot-metrics/mcp-app.html"]
        Tools --> Resources
    end
    
    Resources -->|HTML Bundle| Dashboard
    Server -->|ontoolresult| useApp
    
    subgraph GitHub["GitHub API"]
        API["/enterprises/{slug}/copilot/metrics"]
    end
    
    Server -->|Fetch| API
    API -->|Metrics Data| Server

Data Flow

sequenceDiagram
    participant User
    participant Model as AI Model
    participant Server as MCP Server
    participant API as GitHub API
    participant UI as React Dashboard

    User->>Model: "Generate Copilot report for acme-corp"
    Model->>Server: generate_copilot_report(enterpriseSlug, dateRange)
    Server->>API: GET /enterprises/acme-corp/copilot/metrics
    API-->>Server: CopilotUsageMetrics[]
    Server->>Server: Transform to CopilotReport
    Server-->>Model: JSON Report
    Server-->>UI: ontoolresult(report)
    UI->>UI: Render charts & tables
    Model-->>User: Summary with insights

MCP Integration

Tools

`generate_copilot_report`

Generate a comprehensive Copilot usage report with interactive dashboard.

Parameter	Type	Required	Description
`enterpriseSlug`	string	Yes	Enterprise identifier
`orgName`	string	No	Organization within enterprise
`dateRange.from`	string	Yes	Start date (YYYY-MM-DD)
`dateRange.to`	string	Yes	End date (YYYY-MM-DD)

Example prompt:

Generate a Copilot report for enterprise "acme-corp" from 2024-01-01 to 2024-01-31

`refresh_report` (App-Only)

Same parameters as above, but with visibility: ["app"] — hidden from AI, invocable only by the UI for refresh actions.

Resources

URI	Description
`ui://copilot-metrics/mcp-app.html`	Interactive React dashboard

Usage Examples

Here are some common prompts to get the most out of Copilot Compass:

📸 See example prompt

Basic Report Generation

Generate a Copilot report for enterprise "acme-corp" from 2024-01-01 to 2024-01-31

Monthly Adoption Analysis

Show me Copilot usage for "my-enterprise" for the last 30 days and highlight adoption trends

Organization-Specific Metrics

Generate a Copilot report for organization "engineering-team" within enterprise "acme-corp" for Q4 2024

Comparing Time Periods

Generate Copilot reports for enterprise "acme-corp" for January and February 2024 and compare the adoption rates

Deep Dive Analysis

Analyze the Copilot report for "acme-corp" and tell me:
1. Which languages have the highest acceptance rates?
2. What's the trend in daily active users?
3. Which editors are most popular?
4. How is Copilot Chat being used?

Dashboard Insights

The AI model can analyze the report data and provide insights like:

Adoption Health: Are active users growing, stable, or declining?
Language Hotspots: Which languages benefit most from Copilot?
Feature Usage: Balance between code completions, chat, and PR summaries
Editor Preferences: Which IDEs your team prefers for Copilot interactions

Project Structure

copilot-compass/
├── server.ts                    # MCP server entry (tools + resources)
├── src/
│   ├── types.ts                 # TypeScript interfaces
│   ├── github-client.ts         # GitHub API client with caching
│   ├── report-generator.ts      # Report generation + mock data
│   ├── html-report-generator.ts # Standalone HTML export
│   ├── svg-charts.ts            # Inline SVG for markdown
│   ├── visual-report-generator.ts # Markdown report with charts
│   ├── mcp-app.tsx              # React dashboard entry
│   └── global.css               # Dashboard styles
├── mcp-app.html                 # Vite entry point
├── assets/
│   └── icon.svg                 # Compass icon
├── dist/                        # Build output
├── .env.example                 # Sample environment variables
├── vite.config.ts               # Vite config (singlefile plugin)
├── tsconfig.json                # TypeScript config (React)
└── tsconfig.server.json         # TypeScript config (Node.js)

Configuration

Copy the sample environment file and configure your settings:

cp .env.example .env

Environment Variables

Variable	Description	Default
`GITHUB_TOKEN`	GitHub PAT with Copilot metrics access	—
`PORT`	Server port	`3001`
`MCP_SERVER_PORT`	Alternative port variable	`3001`
`CACHE_TTL_SECONDS`	API response cache duration	`300`

GitHub Token Scopes

Scope	Purpose
`manage_billing:copilot`	Required for metrics access
`read:enterprise`	Enterprise-level queries
`read:org`	Organization-level queries

Development

# Watch mode for UI development
npm run dev

# Build for production
npm run build

# Run server (requires build first)
npm start

# Type checking
npx tsc --noEmit

Testing

The project includes a comprehensive test suite using Vitest.

Running Tests

# Run all tests
npm test

# Run tests in watch mode (re-runs on file changes)
npm run test:watch

# Run tests with coverage report
npm run test:coverage

Test Coverage

Module	Coverage	Description
`github-client.ts`	98%	API client, caching, error handling
`report-generator.ts`	98%	Report generation, mock data, aggregation
`svg-charts.ts`	100%	SVG chart generation functions
`schemas.ts`	100%	Zod schema validation

Test Structure

tests/
├── setup.ts                  # Global test setup and mocks
├── github-client.test.ts     # API client tests (15 tests)
├── report-generator.test.ts  # Report generator tests (18 tests)
├── svg-charts.test.ts        # Chart generation tests (40 tests)
├── integration.test.ts       # End-to-end pipeline tests (37 tests)
└── fixtures/
    └── api-responses.ts      # Realistic GitHub API response fixtures

Total: 110 tests

What's Tested

GitHub API Client: Token validation, enterprise/org endpoints, caching behavior, error handling
Report Generator: Live data transformation, mock data fallback, summary aggregation, edge cases
SVG Charts: Bar, line, donut, horizontal bar, and sparkline chart generation
Schema Validation: Zod schema parsing, API response validation, error formatting
Integration Pipeline: End-to-end data flow with realistic fixtures for both enterprise and organization metrics

API Response Validation

The project uses Zod schemas to validate GitHub API responses at runtime, ensuring data integrity:

// Validation is automatic - errors are thrown if API response is malformed
const metrics = await client.getEnterpriseMetrics('my-enterprise', dateRange);

Validation can be disabled for performance (not recommended):

VALIDATE_API_RESPONSES=false npm start

Troubleshooting

Common Issues

"GitHub API error (403): Resource not accessible"

Cause: Your GitHub token lacks the required scopes.

Solution: Create a new PAT with these scopes:

manage_billing:copilot (required)
read:enterprise for enterprise metrics
read:org for organization metrics

"GitHub API error (404): Not Found"

Cause: The enterprise slug or organization name doesn't exist, or you don't have access.

Solution:

Verify the enterprise/org name in GitHub settings
Ensure your account has Copilot Business/Enterprise access
Check that you're a member of the organization

Dashboard shows "Demo Data" / Mock Data

Cause: The GitHub API call failed, so the app falls back to demo data.

Solution:

Check the server logs for API error messages
Verify your GITHUB_TOKEN environment variable is set
Ensure the token has the correct scopes
Try a shorter date range (API returns max 28 days per request)

Server won't start / Port already in use

Solution:

# Find and kill the process using port 3001
npx kill-port 3001

# Or use a different port
PORT=3002 npm start

"Tool not found" in MCP client

Cause: The MCP client hasn't connected to the server or needs a refresh.

Solution:

Restart the MCP server
Reload/restart your MCP client (VS Code, Claude Desktop)
Check the server URL in your MCP configuration

TypeScript compilation errors

Solution:

# Clean build artifacts and rebuild
rm -rf dist && npm run build

Debug Mode

Enable verbose logging to diagnose issues:

DEBUG=* npm start

FAQ

What GitHub Copilot plans are supported?

Copilot Compass works with:

Copilot Business — Organization-level access
Copilot Enterprise — Enterprise + organization-level access

Individual Copilot Pro subscriptions do not have access to the metrics API.

How far back can I query data?

The GitHub API returns up to 28 days of data per request. For longer periods, the app automatically handles pagination. Historical data is available from when Copilot was enabled for your organization.

Is the API response cached?

Yes, responses are cached for 5 minutes (configurable via CACHE_TTL_SECONDS). This reduces API calls during repeated queries or dashboard refreshes.

Can I run this without a GitHub token?

Yes, for demos. The app falls back to realistic mock data if no token is provided or the API fails. Look for dataSource: "mock" in the report.

What's the difference between enterprise and organization reports?

Enterprise: Aggregated metrics across ALL organizations in your enterprise
Organization: Metrics for a specific organization only

How do I interpret acceptance rate?

≥40%: Excellent — Copilot is significantly boosting productivity
25-40%: Good — Developers are finding value in suggestions
<25%: Review — Consider training or language-specific tuning

Can I export the data?

The JSON report returned by the tool contains all metrics. The AI model can format this into Markdown, CSV, or other formats on request.

Does this work with GitHub Enterprise Server (GHES)?

Currently, only GitHub Cloud (github.com) is supported. GHES support would require modifying the API base URL.

Tech Stack

Runtime: Node.js 18+ with native ES modules
Framework: MCP Apps SDK (@modelcontextprotocol/ext-apps)
UI: React 19 + Chart.js 4 + react-chartjs-2
Build: Vite 6 + vite-plugin-singlefile (bundles to one HTML)
Transport: Streamable HTTP (stateless mode)
Validation: Zod schemas for tool inputs

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Related Resources

GitHub Copilot Documentation

Copilot Metrics API Reference — Official API documentation
About GitHub Copilot Metrics — Understanding the metrics
Copilot Business vs Enterprise — Plan comparison

MCP Resources

MCP Apps SDK — SDK for building MCP apps with UIs
Model Context Protocol — Protocol specification
MCP Servers Repository — Community MCP servers

Technologies Used

Chart.js — Charting library
Zod — TypeScript-first schema validation
Vite — Frontend build tool
Vitest — Unit test framework

License

MIT - Samuel Tauil

Built with the MCP Apps SDK

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.devcontainer		.devcontainer
.github		.github
.vscode		.vscode
assets		assets
docs		docs
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
mcp-app.html		mcp-app.html
package-lock.json		package-lock.json
package.json		package.json
server.ts		server.ts
tsconfig.json		tsconfig.json
tsconfig.server.json		tsconfig.server.json
vite.config.ts		vite.config.ts
vitest.config.ts		vitest.config.ts

Folders and files

Latest commit

History

Repository files navigation

Copilot Compass

Overview

What Makes This Special?

Features

Analytics Dashboard

Understanding the Metrics

Report Generation

MCP Apps SDK Features

Quick Start

Prerequisites

Installation

Build & Run

Connect Your MCP Client

Run in GitHub Codespaces

Option A: Connect from VS Code Insiders Desktop (Full UI)

Option B: Use Codespaces Web Editor (Tools Only)

What Happens Automatically

Configure GitHub Token (Optional)

Verify It's Working

Manual Start (If Needed)

Public Access via Tunnel (Local Development)

Option 1: Cloudflare Tunnel (Recommended)

Option 2: ngrok

Option 3: Tailscale Funnel

Running as a Background Service

Architecture

Data Flow

MCP Integration

Tools

generate_copilot_report

refresh_report (App-Only)

Resources

Usage Examples

Basic Report Generation

Monthly Adoption Analysis

Organization-Specific Metrics

Comparing Time Periods

Deep Dive Analysis

Dashboard Insights

Project Structure

Configuration

Environment Variables

GitHub Token Scopes

Development

Testing

Running Tests

Test Coverage

Test Structure

What's Tested

API Response Validation

Troubleshooting

Common Issues

"GitHub API error (403): Resource not accessible"

"GitHub API error (404): Not Found"

Dashboard shows "Demo Data" / Mock Data

Server won't start / Port already in use

"Tool not found" in MCP client

TypeScript compilation errors

Debug Mode

FAQ

What GitHub Copilot plans are supported?

How far back can I query data?

Is the API response cached?

Can I run this without a GitHub token?

What's the difference between enterprise and organization reports?

How do I interpret acceptance rate?

Can I export the data?

Does this work with GitHub Enterprise Server (GHES)?

Tech Stack

Contributing

Related Resources

GitHub Copilot Documentation

MCP Resources

Technologies Used

License

About

`generate_copilot_report`

`refresh_report` (App-Only)

Packages