Argocd gitops dev phase 1

.claude/skills/check-alerts/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: check-alerts
+description: Check currently firing Grafana alerts, analyze alert status, and investigate alert issues in the Kagenti platform
+---
+
+# Check Alerts Skill
+
+This skill helps you check and analyze Grafana alerts in the Kagenti platform.
+
+## When to Use
+
+- User asks "what alerts are firing?"
+- User wants to check alert status
+- After platform changes or deployments
+- During incident investigation
+- When troubleshooting platform issues
+
+## What This Skill Does
+
+1. **List Firing Alerts**: Show all currently active alerts
+2. **Alert Details**: Display alert severity, component, and description
+3. **Alert History**: Check recent alert state changes
+4. **Query Alert Rules**: Verify alert configuration
+5. **Test Alert Queries**: Validate PromQL queries
+
+## Examples
+
+### Check Firing Alerts
+
+```bash
+# Get all currently firing alerts from Grafana
+kubectl exec -n observability deployment/grafana -- \
+  curl -s 'http://localhost:3000/api/alertmanager/grafana/api/v2/alerts' \
+  -u admin:admin123 | python3 -c "
+import sys, json
+alerts = json.load(sys.stdin)
+firing = [a for a in alerts if a.get('status', {}).get('state') == 'active']
+print(f'Firing alerts: {len(firing)}')
+for alert in firing:
+    labels = alert.get('labels', {})
+    annotations = alert.get('annotations', {})
+    print(f\"\\n• {labels.get('alertname')} ({labels.get('severity')})\")
+    print(f\"  Component: {labels.get('component')}\")
+    print(f\"  Description: {annotations.get('description', 'N/A')[:100]}...\")
+"
+```
+
+### List All Alert Rules
+
+```bash
+# Get all configured alert rules
+kubectl exec -n observability deployment/grafana -- \
+  curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \
+  -u admin:admin123 | python3 -c "
+import sys, json
+rules = json.load(sys.stdin)
+print(f'Total alert rules: {len(rules)}')
+for rule in rules:
+    print(f\"  • {rule.get('title')} ({rule.get('labels', {}).get('severity')})\")
+"
+```
+
+### Check Specific Alert Configuration
+
+```bash
+# Get configuration for a specific alert
+kubectl exec -n observability deployment/grafana -- \
+  curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \
+  -u admin:admin123 | python3 -c "
+import sys, json
+rules = json.load(sys.stdin)
+alert_uid = 'prometheus-down'  # Change this to the alert UID
+rule = next((r for r in rules if r.get('uid') == alert_uid), None)
+if rule:
+    print(f\"Alert: {rule.get('title')}\")
+    print(f\"Query: {rule.get('data', [{}])[0].get('model', {}).get('expr')}\")
+    print(f\"noDataState: {rule.get('noDataState')}\")
+    print(f\"execErrState: {rule.get('execErrState')}\")
+"
+```
+
+### Test Alert Query Against Prometheus
+
+```bash
+# Test an alert's PromQL query
+QUERY='up{job="kubernetes-pods",app="prometheus"} == 0'
+
+kubectl exec -n observability deployment/grafana -- \
+  curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query' \
+  --data-urlencode "query=${QUERY}" | python3 -m json.tool
+```
+
+### Check Alert Evaluation State
+
+```bash
+# Check why an alert is firing or not firing
+kubectl exec -n observability deployment/grafana -- \
+  curl -s 'http://localhost:3000/api/v1/eval/rules' \
+  -u admin:admin123 | python3 -m json.tool
+```
+
+## Alert Locations in Grafana UI
+
+**Access Grafana**: https://grafana.localtest.me:9443
+**Credentials**: admin / admin123
+
+**Navigation**:
+1. **Alerting** → **Alert rules** - View all configured alerts
+2. **Alerting** → **Alert list** - See firing/pending alerts
+3. **Alerting** → **Silences** - Manage alert silences
+4. **Alerting** → **Contact points** - Check notification settings
+5. **Alerting** → **Notification policies** - View routing rules
+
+## Common Alert Issues
+
+### False Positives
+- Check `noDataState` configuration (should be `OK` for most alerts)
+- Verify query matches actual resource type (Deployment vs StatefulSet)
+- Test query returns correct results
+
+### Alert Not Firing When It Should
+- Verify metric exists in Prometheus
+- Check alert threshold is appropriate
+- Verify `for` duration isn't too long
+- Check `noDataState` isn't masking the issue
+
+### Alert Configuration Not Loading
+- Restart Grafana: `kubectl rollout restart deployment/grafana -n observability`
+- Check ConfigMap applied: `kubectl get configmap grafana-alerting -n observability`
+- Verify no YAML syntax errors
+
+## Related Documentation
+
+- [Alert Runbooks](../../../docs/runbooks/alerts/)
+- [Alert Testing Guide](../../../docs/04-observability/ALERT_TESTING_GUIDE.md)
+- [CLAUDE.md Alert Monitoring](../../../CLAUDE.md#alert-monitoring)
+- [TODO_INCIDENTS.md](../../../TODO_INCIDENTS.md) - Current incident tracking
+
+## Runbooks by Alert
+
+When an alert fires, consult its runbook:
+- `docs/runbooks/alerts/<alert-uid>.md`
+
+Example: If "Prometheus Down" alert fires → `docs/runbooks/alerts/prometheus-down.md`
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)