Collect, visualize, expose, and alert on table/index sizes, growth, usage, and locking (#1103)#1114
Merged
erikdarlingdata merged 5 commits intoJun 12, 2026
Conversation
Collection layer for both apps. A daily collector captures per-table and per-index size, growth, index usage, and locking/contention from sys.dm_db_partition_stats, sys.dm_db_index_usage_stats, and sys.dm_db_index_operational_stats (stable 2016+, Azure SQL DB, and MI). Dashboard: collect.index_object_stats table (02 + 06 ensure block + NC index), collector proc (55), schedule row + master dispatch (04/42). Dynamic retention (43) picks the table up automatically by collection_time. Lite: RemoteCollectorService.IndexObjectStats.cs (per-DB cross-database sp_executesql on-prem; per-database connect on Azure), DuckDB schema v29 + archival registration in BOTH ArchivableTables arrays, schedule default. Cumulative usage/locking counters carry sqlserver_start_time as the reset boundary so deltas are computed safely in the read layer. Verified live on SQL 2016: 148-149 rows across 6 databases, unused/write-only index signal works. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read layer (both apps): GetObjectSizeGrowthAsync (per-table size + 7d/30d growth + daily rate, indexes rolled up), GetIndexUsageAsync (seeks/scans/ lookups/updates + Unused/Write-only/Active classification), GetIndexLockingAsync (row/page lock waits, escalations, latch waits, top contended). Dashboard reads collect.index_object_stats; Lite reads v_index_object_stats. Dashboard growth queries validated live on SQL 2016. UI: three FinOps sub-tabs in both apps - "Object Sizes & Growth", "Index Usage", "Locking & Contention" - mirroring the existing Storage Growth grid pattern (sortable columns, column filters on identity columns, refresh, count indicators). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three read tools in both apps exposed to the AI surface: get_table_index_sizes (largest tables + growth), get_index_usage (with Unused/Write-only/Active classification), get_object_locking (lock/latch waits + escalations, top contended). Registered in McpHostService, documented in McpInstructions under a new "Storage & Index Tools" section. McpSchemaCompatTests pass (3/3 each app). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daily object/index data is cumulative, so detection is delta-based (two most
recent snapshots) rather than stddev-baseline. New detector method in both
AnomalyDetector (Lite/DuckDB) and SqlServerAnomalyDetector (Dashboard/SQL):
- ANOMALY_OBJECT_GROWTH: biggest day-over-day table grower over 100 MB + 20%.
- ANOMALY_OBJECT_CONTENTION: index with largest new row-lock wait time (>=60s),
guarded against counter resets (cur.ms >= prv.ms).
Joins on stable object_id/index_id; emits one fact each (worst offender) to avoid
notification spam. FactScorer scores both (ratio-based); FactAdvice supplies
headline/investigation/remediation. Findings flow through the existing
AnalysisNotificationService path (severity gate + cooldown) — no email/settings
plumbing needed. Growth detection validated live on SQL 2016 with a synthetic
prior snapshot (Posts 2278->6835 MB, 200%, detected + cleaned up).
Thresholds are currently constants (not yet user-configurable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…style (#1103) Tests (Lite.Tests/IndexObjectStatsTests.cs): seed two daily snapshots and exercise the DuckDB read layer (date_diff growth math, GREATEST last-access, unused/write-only classification, contention rows) and the delta-based anomaly detector (growth + contention fire; single-snapshot yields no false positives). Bump DuckDbSchemaTests table count 30->31 for the new index_object_stats table. Style (per T-SQL review): reformat the embedded queries in SqlServerAnomalyDetector.DetectObjectStatsAnomalies and two scalar subqueries in DatabaseService.FinOps.IndexObjects to house style (multi-line CTEs, one column per line, JOIN/ON layout with correlation order). No behavior change. Full suites green: Lite 434, Dashboard 482. Correctness + security self-review clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1103.
Adds time-series collection + visualization + MCP + alerting for per-table / per-index size, growth, index usage, and locking/contention, across both Dashboard and Lite. All from stock DMVs (
sys.dm_db_partition_stats,sys.dm_db_index_usage_stats,sys.dm_db_index_operational_stats), verified stable 2016 → 2025 / Azure SQL DB / MI.What's included (6 commits, by layer)
install/55_collect_index_object_stats.sql(cross-DB cursor + Azure EngineEdition=5 branch) →collect.index_object_stats(02/06, NC index), scheduled daily / 90-day (04/42); dynamic retention picks it up. LiteRemoteCollectorService.IndexObjectStats.cs→ DuckDB schema v29 + archival in bothArchivableTablesarrays + schedule.GetObjectSizeGrowthAsync(per-table size + 7d/30d growth + daily rate),GetIndexUsageAsync(Unused / Write-only / Active classification),GetIndexLockingAsync(top contended) in both apps.get_table_index_sizes,get_index_usage,get_object_lockingin both apps.ANOMALY_OBJECT_GROWTH(table grew >100 MB + 20% day-over-day) andANOMALY_OBJECT_CONTENTION(index gained ≥60s new row-lock wait, reset-guarded). Flows through the existing anomaly→AnalysisNotificationServicepath;FactScorer+FactAdviceextended.Verification done
IndexObjectStatsTests(5) cover the Lite DuckDB read dialect + the anomaly detector (incl. single-snapshot no-false-positive). MCP schema-compat tests pass (3/3 each).Open items for follow-up (flagged, not silently dropped)
IAlertSettings).topNas a DuckDB parameter in the Lite read layer (currently a safeintinterpolation) for defense-in-depth.🤖 Generated with Claude Code