Skip to content

PLAN.md

Latest commit

 

History

History
156 lines (136 loc) · 9.37 KB

PLAN.md

File metadata and controls

156 lines (136 loc) · 9.37 KB
Raw

Plan: LLM Monitoring → Triage → Ticket → PR (POC)

Context

A client-facing POC proving an LLM can autonomously run an end-to-end ops workflow: read monitoring logs → triage errors → file a ticket → write a code fix → open a PR, with a human only firing the initial trigger.

Constraints: ~1.5 business days, lightweight MVP we can iterate on.

Locked decisions:

  • Orchestrator: Python
  • Ticketing: Trello
  • Fix/PR (Phase 1): single-shot patch generated by Claude, committed to a branch, PR opened via the gh CLI. Behind a Fixer interface so an Agent SDK implementation drops in later with no orchestrator changes.
  • Target codebase: a self-contained planted-bug sample_service/ in this repo; the PR opens against this same repo.

Environment verified: gh authed as paytown; origin = github.com/stride-nyc/llm-monitoring-demo; .env holds CLAUDE_API_KEY.

Architecture

A linear pipeline orchestrated by one entrypoint. Every external system sits behind a small module so the mock version now and the real version later share an interface.

run.py                      # CLI entrypoint: `python run.py --since 1d`
agent/
  config.py                 # load .env (reads CLAUDE_API_KEY), settings, FIXER flag
  llm.py                    # Anthropic SDK wrapper; prompt caching; structured tool-use helper
  monitoring.py             # read_alerts(since) -> list[LogEntry]  (mock JSON now; Sentry/DataDog later)
  triage.py                 # triage(entries) -> list[TriageResult] via Claude
  ticketing.py              # Trello: create_card -> Ticket; move_card(card, list); attach PR link
                            #   card state machine: To Do -> Doing -> PR (live on the board)
  fixer.py                  # Fixer protocol + SingleShotFixer (now), AgentSDKFixer (stub)
  github_pr.py              # branch + commit patched file + `gh pr create` -> pr_url
  models.py                 # dataclasses: LogEntry, TriageResult, Ticket, FixResult
mock_data/logs.json         # timestamped alerts; error entries point at sample_service bug(s)
sample_service/
  app.py                    # tiny buggy "service" the logs reference
  test_app.py               # failing test demonstrating the bug
.env.example                # documents all required keys
requirements.txt            # anthropic, python-dotenv, requests

Flow (run.py orchestration)

  1. Trigger (human-in-the-loop, once): python run.py --since 1d. Narrates each step to stdout.
  2. Read logs: monitoring.read_alerts(since) loads mock_data/logs.json, filters by time window (timestamps relative to today so --since visibly works). Returns LogEntry list.
  3. Triage: triage.triage(entries) sends ERROR-level entries to Claude to dedupe/group, assign severity, identify the implicated file + likely root cause, and produce a ticket title + body. Structured JSON via tool-use (no brittle parsing).
  4. Ticket: for each actionable TriageResult, ticketing.create_card() POSTs a Trello card to the To Do list and returns its URL/id.
  5. Fix → PR: for each ticket:
    • Move card To Do → Doing (ticketing.move_card) to show work has started.
    • fixer.fix(ticket, repo_context): SingleShotFixer reads the implicated file, sends file + ticket + error to Claude, gets the full corrected file, writes it to branch fix/<slug>-<cardid>, commits, and github_pr.open_pr() opens the PR via the GitHub REST API.
    • PR body references the Trello card; attach the PR URL to the card and move it Doing → PR (loop closed both directions; card marches across the board live).
  6. Summary: print alert → ticket URL → PR URL for each handled error.

Fixer abstraction (the staged plan)

Both implementations take identical inputs and produce identical outputs — the only difference is how files on disk get changed. Everything upstream (triage, ticket) and downstream (commit, PR, comment) is untouched.

triage → ticket → [ Fixer.fix(ticket, ctx) → FixResult ] → git commit → gh pr create → Trello comment
                         ^ only this box changes               ^ consumes FixResult; agnostic to how made

class Fixer(Protocol):
    def fix(self, ticket: Ticket, ctx: RepoContext) -> FixResult: ...
    # FixResult = (branch_name, files_changed, summary)

def make_fixer(config) -> Fixer:        # factory keyed on config.FIXER
    return AgentSDKFixer() if config.FIXER == "agent_sdk" else SingleShotFixer()
  • run.py calls make_fixer(config) once, then fixer.fix(...). Never branches on the impl.
  • Phase 1 — SingleShotFixer: one Claude call returns corrected file content; writes one file. Deterministic, demo-safe, fast.
  • Phase 2 — AgentSDKFixer (if time): Claude Agent SDK loop with read/edit/bash tools — can explore, fix multiple files, run tests. Edits the working tree directly.
  • Both leave the same artifact: a changed working tree + a FixResult. Upgrading = add one file
    • flip FIXER=agent_sdk. Zero changes to run.py.

Key implementation notes

  • LLM (llm.py): Anthropic Python SDK, client built with api_key from CLAUDE_API_KEY. claude-opus-4-8 for both triage and fix (best quality for the demo; cost-tune later via the TRIAGE_MODEL / FIX_MODEL env vars). Adaptive thinking + structured outputs (messages.parse) + prompt caching on the system prompt / file context.
  • Planted bug: one clear bug in sample_service/app.py (e.g. KeyError / off-by-one / unguarded division). The matching logs.json error carries a stack trace naming file + line.
  • Time filtering: seed logs.json with several entries inside the last day and a couple older so --since 1d demonstrably filters.
  • Mock-to-real seams: monitoring and fixer are the only two swap points; both isolated.
  • Safety: every PR opens on a fresh branch; nothing auto-merges; human reviews the PR. New .env keys: TRELLO_KEY, TRELLO_TOKEN, TRELLO_LIST_ID (documented in .env.example).

Scaling monitoring to a real service (Phase 2, if MVP lands early)

read_alerts(since) sits behind a MonitoringSource interface; the LogEntry model is the contract. Mock reads JSON; a real adapter maps a provider's API response into LogEntry.

class MonitoringSource(Protocol):
    def read_alerts(self, since: timedelta) -> list[LogEntry]: ...
# MockSource (now) reads mock_data/logs.json
# SentrySource / DatadogSource (later) call the provider API and map -> LogEntry

Recommended provider: Sentry (not Google Analytics):

  • Sentry (★ recommended): purpose-built error tracking. Events carry exception type, message, stack trace + file/line, release — exactly what triage needs. Free tier, token auth. GET /api/0/projects/{org}/{proj}/issues/?statsPeriod=24h then latest event for stack detail.
  • DataDog Logs (alternative): POST /api/v2/logs/events/search with status:error + time window. Strong if the client already runs DataDog; logs often lack a code location. API+APP keys.
  • Google Analytics (not recommended): web/product analytics — no errors/stack traces/code locations, nothing for triage→fix to act on.

Shaping the mock like a Sentry issue (stack frame naming file+line) makes SentrySource a near drop-in.

Build order (fits ~1.5 days)

  1. Scaffold: requirements.txt, agent/ package, config.py, models.py, .env.example.
  2. sample_service/app.py + planted bug + mock_data/logs.json referencing it.
  3. monitoring.read_alerts + time-window filtering.
  4. llm.py + triage.py (structured outputs via messages.parse).
  5. ticketing.py Trello create-card + move across board (To Do/Doing/PR).
  6. fixer.py SingleShotFixer + github_pr.py (branch + single commit + PR via REST API).
  7. run.py wiring + narrated output; --dry-run / --max-tickets; end-to-end verified.

Nice-to-haves / follow-ups (post-MVP)

  • Dedicated bot identity for PRs. PRs are currently authored by the token owner (the GITHUB_TOKEN in .env). Create a separate GitHub machine account, mint its PAT, and use that so PRs + commits show as e.g. monitoring-agent-bot — the clean "an agent did this, not a human" look.
  • Real monitoring source (SentrySource / DatadogSource) behind the existing MonitoringSource interface — Sentry recommended (stack traces with file/line map cleanly to triage).
  • AgentSDKFixer — swap the single-shot fix for an agentic Claude Agent SDK loop (multi-file fixes, can run the tests) via FIXER=agent_sdk. Interface already in place.
  • More varied mock logs / multiple simultaneous bugs; a polished demo script.

Verification

  • Unit-ish: run sample_service/test_app.py before fix (fails) / after fix (passes).
  • Per-stage: python run.py --since 1d --dry-run prints parsed logs, triage JSON, and the proposed patch without touching Trello/GitHub.
  • End-to-end: python run.py --since 1d → Trello card created, fix/* branch pushed, PR on github.com/stride-nyc/llm-monitoring-demo, PR URL commented back on the card. --since 1h filters out older seeded errors.

Open setup item for the user

Create a Trello API key + token and a target list, then add TRELLO_KEY / TRELLO_TOKEN / TRELLO_LIST_ID to .env. (If Trello signup is a timeline risk, ticketing can fall back to GitHub Issues with a one-line config change.)