Skip to content

Commit 907c1b6

Browse files
DeepfreezechillBrian KrafftCopilot
authored
Epic 6.2: SLOs - Error Budgets, Burn Rates, Multi-Window Alerting (#51)
* feat: Epic 6.2 - SLO module with error budgets, burn rates, and MCP tool - SLOTarget dataclass with validation, ErrorBudget with float-tolerant exhaustion - BurnRateCalculator with Google SRE multi-window alerting model - SLOEvaluator reads from MetricsRegistry via Prometheus .collect() API - 3 default targets: latency_p99 <30s, error_rate <5%, availability >99% - check_slos MCP tool (8 total tools) - 35+ new tests (1,858 total passed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address all review findings - 5 agents, 12 fixes, 22 new tests Security: - Wrap check_slos in try/except + handle_mcp_exception (error oracle) - Clamp burn_rate to MAX_BURN_RATE=1000.0 (no float('inf') in JSON) - Add SLOTarget validation: non-empty name, finite non-negative threshold - Filter _get_request_counts to _total suffix (PROMETHEUS_ENABLE_CREATED_SERIES safety) Correctness: - Fix objective=1.0 contradictory state (exhausted + 100% remaining) - Fix p99 multi-label aggregation (sum by le across agent × status) - Add p99 linear interpolation between bucket boundaries - Fix p99 overflow: return highest bucket boundary instead of None - Correct docstring: simplified burn-rate ratio, not Google multi-window Hardening: - DEFAULT_TARGETS immutable tuple (was mutable list) - SLOTarget frozen=True dataclass - ErrorBudget zero-budget path consistent status Tests: 22 new adversarial tests (1,884 total passed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: R2 hardening - NaN-safe JSON, isfinite guard, coupling docs - json.dumps(allow_nan=False) prevents NaN/Infinity in JSON output - math.isfinite() guard on p99 interpolation result - ValueError/OverflowError catch in _get_request_counts - Cross-reference comment for status label coupling (execution.py) R2 review: 2 APPROVE (sec, opus), 2 REQUEST_CHANGES on documented design limitations (in-process counters, hardcoded dispatch) - v2 items. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Brian Krafft <bkrafft@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 9a6acd2 commit 907c1b6

9 files changed

Lines changed: 1156 additions & 16 deletions

File tree

RUNBOOK.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
schema_version: '1.0'
22
lock:
3-
active_claim: null
4-
claimed_by: null
5-
claimed_at: null
3+
active_claim: '6.2'
4+
claimed_by: 2026-04-07-e9b4572b
5+
claimed_at: '2026-04-07T00:29:00Z'
66
review:
77
max_rounds: 3
88
required_gates:
@@ -547,8 +547,8 @@ epics:
547547
- id: '6.2'
548548
phase: P6
549549
title: SLOs (latency, error rate, availability)
550-
status: pending
551-
branch: null
550+
status: claimed
551+
branch: epic/6.2-slos
552552
pr: null
553553
review_round: 0
554554
description: 'Define and enforce SLO targets: p99 execution latency, error rate

openspace/mcp/tool_handlers.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -829,6 +829,25 @@ async def get_execution_traces(limit: int = 5) -> str:
829829
return json.dumps([t.to_dict() for t in traces], indent=2, default=str)
830830

831831

832+
async def check_slos() -> str:
833+
"""Check current SLO (Service Level Objective) compliance.
834+
835+
Returns current status for all SLO targets including error budget
836+
remaining, burn rate, and any active alerts. Use this to monitor
837+
system health against defined reliability targets.
838+
"""
839+
try:
840+
from openspace.observability.metrics import metrics
841+
from openspace.observability.slos import SLOEvaluator
842+
843+
evaluator = SLOEvaluator(registry=metrics)
844+
return evaluator.to_json()
845+
except Exception as e:
846+
from openspace.errors import EXECUTION_ERROR, handle_mcp_exception
847+
848+
return handle_mcp_exception(e, tool_name="check_slos", error_code=EXECUTION_ERROR)
849+
850+
832851
# ---------------------------------------------------------------------------
833852
# Registration
834853
# ---------------------------------------------------------------------------
@@ -845,3 +864,5 @@ def register_handlers(mcp) -> None:
845864
mcp.tool()(health_check)
846865
mcp.tool()(get_metrics)
847866
mcp.tool()(get_execution_traces)
867+
# SLOs (Epic 6.2)
868+
mcp.tool()(check_slos)
Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,15 @@
1-
"""openspace.observability — Metrics, tracing, and health for OpenSpace.
1+
"""openspace.observability — Metrics, tracing, health, and SLOs for OpenSpace.
22
33
Provides Prometheus-compatible metrics, structured execution traces,
4-
and aggregated health checks for the MCP server and grounding engine.
4+
aggregated health checks, and SLO tracking for the MCP server and
5+
grounding engine.
56
6-
Epic 6.1 — Phase 6 (Operability).
7+
Epics 6.1–6.2 — Phase 6 (Operability).
78
"""
89

910
from openspace.observability.health import HealthAggregator, HealthStatus
1011
from openspace.observability.metrics import MetricsRegistry
12+
from openspace.observability.slos import SLOEvaluator
1113
from openspace.observability.tracing import ExecutionTracer, trace_async
1214

1315
__all__ = [
@@ -16,4 +18,5 @@
1618
"trace_async",
1719
"HealthAggregator",
1820
"HealthStatus",
21+
"SLOEvaluator",
1922
]

0 commit comments

Comments
 (0)