llm-monitoring-demo

A POC showing an LLM run an end-to-end ops workflow, with a human only in the loop to start it:

read Sentry errors → triage with Claude → open a Trello ticket → write a code fix → open a GitHub PR → assign the Sentry issue

Services

Service	Role
Sentry	Source of truth for errors. The agent reads unresolved issues and, once a fix PR is open, assigns the issue.
Trello	Ticketing. A card is created and moves across the board (To Do → Doing → PR) as the agent works.
GitHub	The fix lands as a pull request against main (branch-protected — the agent can only ever open a PR, never push).

Triage and the code fix are generated by Claude (claude-opus-4-8).

Prerequisites

• Python 3.11+
• A Claude API key, plus tokens for Sentry, Trello, and GitHub (see .env.example)

Setup

# 1. One-command setup: checks Python, creates the venv, installs dependencies
./setup.sh
source .venv/bin/activate

# 2. Create your .env from the template, then paste in your keys
cp .env.example .env

# 3. Verify the environment is healthy (should be ALL GREEN)
pytest test_smoke.py

Important: the virtual environment must be active for python run.py / pytest to work. Run source .venv/bin/activate in every new terminal session (you'll see (.venv) in your prompt when it's active). If you skip it, commands run against system Python and fail with ModuleNotFoundError.

.env keys, grouped by service (setup steps for each are in .env.example):

Keys	For
CLAUDE_API_KEY	Triage + fix generation
SENTRY_AUTH_TOKEN, SENTRY_ORG_SLUG, SENTRY_PROJECT_SLUG, SENTRY_ASSIGNEE	Read errors + assign the resolved issue
TRELLO_KEY, TRELLO_TOKEN, TRELLO_TODO_LIST_ID, TRELLO_DOING_LIST_ID, TRELLO_PR_LIST_ID	Ticketing board
GITHUB_TOKEN, GITHUB_REPO, GITHUB_BASE_BRANCH	Opening the PR

Usage

One command — the only human step:

python run.py --since 1d                  # full run: triage → Trello card → fix → PR → assign
python run.py --since 1d --dry-run         # read + triage + generate the fix; NO external writes
python run.py --since 1d --max-tickets 1   # act on at most N tickets this run

--since accepts 1d, 12h, 30m, 1d12h, etc. A run reads unresolved Sentry issues in the window, acts only on errors in your service code (sample_service/) — other signals are read for context but never ticketed — triages them into tickets with Claude, opens a Trello card (sliding it To Do → Doing → PR), generates a fix, opens a PR against main, links it on the card, and assigns the Sentry issue to SENTRY_ASSIGNEE.

Project layout

agent/            # pipeline stages: config, monitoring (Sentry), triage, ticketing (Trello), fixer, github_pr
sample_service/   # the service whose errors the agent triages and fixes
run.py            # CLI entrypoint that orchestrates the workflow
seed_demo.py      # injects sample events into Sentry for the demo