Agentic SRE — Autonomous Incident Response Agent

An LLM-driven incident response agent built with LangGraph. It ingests alerts from Datadog, pulls correlated metrics/logs/traces/events, diagnoses root cause, plans a remediation, and — within configurable safety guardrails — executes fixes against Kubernetes directly (pod restarts, rollbacks, scaling). The agent's own actions are instrumented with OpenTelemetry (OTLP) and exported to Datadog APM, so the agent is itself observable in the same dashboards it's protecting.

This is a portfolio-grade reference implementation: the control flow, guardrails, and integration surfaces are real and functional; production use requires supplying live credentials and hardening the RBAC/approval policy for your environment.

Architecture

flowchart TD
    A[Datadog Monitor Webhook] --> B[ingest_alert]
    B --> C[gather_telemetry\nmetrics/logs/traces/events]
    C --> D[diagnose\nLLM root-cause analysis]
    D --> E[plan_remediation\nLLM + runbook retrieval]
    E --> F{risk level}
    F -- low/auto --> G[execute_remediation\nK8s API]
    F -- medium/high --> H[request_approval\nSlack]
    H -- approved --> G
    H -- denied/timeout --> K[notify: manual intervention]
    G --> I[verify_fix\nre-query telemetry]
    I -- resolved --> J[notify_resolution + postmortem]
    I -- not resolved --> D

Loading

How it works

1. Ingest — POST /webhooks/datadog receives a Datadog monitor notification and normalizes it into an Alert.
2. Gather telemetry — queries the Datadog API for metrics, logs, APM traces, and events scoped to the affected service/pod/tags around the alert window.
3. Diagnose — an LLM (Claude by default) reasons over the telemetry snapshot and produces a structured root-cause hypothesis with a confidence score and supporting evidence.
4. Plan remediation — the agent matches the alert against a local runbook library (src/agentic_sre/runbooks/library/*.yaml) to retrieve candidate actions, then asks the LLM to select/refine a remediation plan and assign a risk level (low, medium, high).
5. Guardrail gate — actions at or below AUTO_REMEDIATE_MAX_RISK execute automatically. Anything riskier is posted to Slack with Approve/Deny and the graph pauses (LangGraph interrupt) until a human responds.
6. Execute — the Kubernetes tool performs the action (restart deployment, scale replicas, rollback to previous revision) using the official kubernetes Python client.
7. Verify — telemetry is re-queried after a cooldown; if the signal hasn't recovered, the graph loops back into diagnosis (bounded by MAX_DIAGNOSIS_RETRIES) instead of blindly repeating the same fix.
8. Notify + postmortem — a Slack/PagerDuty summary is sent and the LLM drafts a short markdown postmortem attached to the incident record.

Project layout

src/agentic_sre/
  agent/           LangGraph state machine (state.py, nodes.py, graph.py, prompts.py)
  tools/           Datadog, Kubernetes, and notification integrations
  otel/            OpenTelemetry instrumentation (traces/metrics/logs -> OTLP -> Datadog)
  runbooks/        YAML runbook library + retrieval logic
  config.py        Settings (env-driven)
  models.py        Pydantic domain models (Alert, Diagnosis, RemediationAction, Incident)
  main.py          FastAPI app (webhook receiver + incident status API)
docker/            Dockerfile, docker-compose (agent + otel-collector), collector config
k8s/               Deployment, least-privilege RBAC, Service
tests/             Unit tests (mocked Datadog/K8s clients)

Setup

cp .env.example .env    # fill in DD_API_KEY, DD_APP_KEY, ANTHROPIC_API_KEY, etc.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn src.agentic_sre.main:app --reload --port 8080

Point a Datadog webhook integration at http://<host>:8080/webhooks/datadog (include $AGENT_WEBHOOK_SECRET as a query param or header per your Datadog webhook config) and Datadog monitors will start triggering agent runs.

Run with Docker Compose (agent + OTel Collector)

docker compose -f docker/docker-compose.yml up --build

This starts the agent alongside an otel/opentelemetry-collector-contrib instance pre-configured (docker/otel-collector-config.yaml) to forward OTLP traces/metrics/logs to Datadog via the collector's native Datadog exporter.

Deploy to Kubernetes

kubectl apply -f k8s/rbac.yaml
kubectl apply -f k8s/deployment.yaml
kubectl apply -f k8s/service.yaml

k8s/rbac.yaml grants the agent's ServiceAccount only what it needs: read pods/events, and get/patch/update on Deployments in its namespace — no cluster-admin, no secrets access.

Environment variables

Variable	Purpose
ANTHROPIC_API_KEY / LLM_MODEL	LLM used for diagnosis and remediation planning
DD_API_KEY / DD_APP_KEY / DD_SITE	Datadog API access for telemetry queries
OTEL_EXPORTER_OTLP_ENDPOINT	Where the agent sends its own traces/metrics/logs
K8S_IN_CLUSTER / KUBECONFIG / K8S_NAMESPACE	Kubernetes client configuration
SLACK_WEBHOOK_URL / PAGERDUTY_ROUTING_KEY	Notification + approval channel
AUTO_REMEDIATE_MAX_RISK	Highest risk level the agent may act on without human approval
MAX_DIAGNOSIS_RETRIES	Bound on diagnose→remediate→verify loop before escalating
AGENT_WEBHOOK_SECRET	Shared secret validated on inbound Datadog webhooks

See .env.example for the full list.

Safety model

• Every remediation action carries a risk_level; only AUTO_REMEDIATE_MAX_RISK and below run unattended.
• Higher-risk actions require explicit human approval via Slack before the Kubernetes tool is invoked.
• The verify step never assumes success — it re-queries telemetry and will retry diagnosis rather than repeat a failed action indefinitely.
• RBAC is scoped to the minimum verbs needed (no delete, no secrets, namespace-scoped).

Extending

• New failure modes: drop a YAML file into src/agentic_sre/runbooks/library/ describing match conditions and candidate actions — no code changes required.
• New remediation actions: add a method to tools/kubernetes_tool.py and reference its name in a runbook or the planning prompt.
• Different LLM provider: swap the model client constructed in agent/nodes.py (langchain-anthropic by default; langchain-openai etc. are drop-in compatible with LangGraph).

License

MIT — see LICENSE.