Feature/added more sql server checks

docs/samples/CheckMSSQL_check_mssql_availability_groups_desc.md

@@ -0,0 +1,46 @@
+#### About `check_mssql_availability_groups`
+
+`check_mssql_availability_groups` reports **Always On availability group
+health** from `sys.dm_hadr_availability_replica_states` and
+`sys.dm_hadr_database_replica_states`, producing one row per replica and per
+availability database on each replica. Replicas that fall out of
+synchronisation silently break the RPO the cluster was built for — this check
+makes that state page before a failover discovers it.
+
+Keywords (one row per replica or replica database):
+
+| Keyword           | Description                                                                  |
+|-------------------|-------------------------------------------------------------------------------|
+| `name`            | `group/replica` or `group/replica/database`                                  |
+| `group`           | Availability group name                                                       |
+| `replica`         | Replica server name                                                           |
+| `role`            | `PRIMARY`, `SECONDARY` or `RESOLVING` (no primary, e.g. failover in progress) |
+| `connected_state` | `CONNECTED` or `DISCONNECTED`                                                 |
+| `replica_health`  | Replica synchronization health: `HEALTHY`, `PARTIALLY_HEALTHY`, `NOT_HEALTHY` |
+| `health`          | Effective health: database health on database rows, replica health otherwise  |
+| `database`        | Availability database name (empty on replica-level rows)                     |
+| `sync_state`      | `SYNCHRONIZED`, `SYNCHRONIZING`, `NOT SYNCHRONIZING`, ...                     |
+| `db_health`       | Database synchronization health (empty on replica-level rows)                |
+| `redo_queue`      | Log received but not yet applied on the secondary, in bytes — failover/RTO lag (accepts units) |
+| `log_send_queue`  | Log not yet sent to the secondary, in bytes — potential data loss/RPO lag (accepts units) |
+| `is_suspended`    | `1` if data movement for the database is suspended                            |
+| `is_local`        | `1` if the row describes the instance being checked                           |
+
+Defaults: **WARNING** on `PARTIALLY_HEALTHY`, **CRITICAL** on `NOT_HEALTHY`,
+`DISCONNECTED`, suspended data movement or a `RESOLVING` role. The health
+states already encode Microsoft's own policy evaluation, so the defaults catch
+broken replication without tuning; add `redo_queue`/`log_send_queue`
+thresholds to alert on lag *before* it degrades health, sized to your RPO/RTO.
+
+**Where to run it:** the primary sees the state of every replica including the
+send/redo queues of all secondaries — pointing this check at the AG listener
+or the primary gives the full picture. A secondary only exposes its local
+replica state (remote replicas without state rows are deliberately omitted
+rather than misreported as DISCONNECTED).
+
+empty-state is **OK** (`No availability groups found`) so the check can be
+rolled out fleet-wide, including instances without AGs. On hosts where an AG
+**must** exist, set `empty-state=critical`: a dropped AG silently removes the
+protection it provided.
+
+Rights: `VIEW SERVER STATE`.

docs/samples/CheckMSSQL_check_mssql_availability_groups_samples.md

@@ -0,0 +1,42 @@
+**Default check (healthy AG):**
+
+```
+check_mssql_availability_groups
+OK: All 1 availability replicas/databases are healthy
+```
+
+**Data movement suspended (or any NOT_HEALTHY state) — critical by default:**
+
+```
+check_mssql_availability_groups
+CRITICAL: 1/1 availability replicas/databases (ag1/f80785925e72/agdb: NOT_HEALTHY)
+```
+
+**Show role, connection and synchronization state of every replica/database:**
+
+```
+check_mssql_availability_groups "warning=none" "critical=health = 'NOT_HEALTHY'" "top-syntax=${status}: ${list}" "detail-syntax=${name}: ${role} ${connected_state} ${health} (${sync_state})"
+OK: ag1/f80785925e72/agdb: PRIMARY CONNECTED HEALTHY (SYNCHRONIZED)
+```
+
+**Alert on replication lag before it breaks the RPO/RTO (size units):**
+
+```
+check_mssql_availability_groups "warning=redo_queue > 500M" "critical=log_send_queue > 1G"
+OK: All 1 availability replicas/databases are healthy|'ag1/f80785925e72/agdb_log_send_queue'=0B;0;1073741824 'ag1/f80785925e72/agdb_redo_queue'=0B;524288000;0
+```
+
+**No availability groups configured — OK by default, so the check can be
+deployed fleet-wide:**
+
+```
+check_mssql_availability_groups
+OK: No availability groups found
+```
+
+**On a host where an AG must exist, make its absence page:**
+
+```
+check_mssql_availability_groups empty-state=critical
+CRITICAL: No availability groups found
+```

docs/samples/CheckMSSQL_check_mssql_blocking_desc.md

@@ -0,0 +1,43 @@
+#### About `check_mssql_blocking`
+
+`check_mssql_blocking` reports **currently blocked sessions** from
+`sys.dm_exec_requests` (`blocking_session_id <> 0`), producing one row per
+blocked request. Blocking chains are the most common "the application is
+frozen" root cause on Windows application stacks, and this check points
+straight at the session everyone is waiting on.
+
+Keywords (one row per blocked request):
+
+| Keyword               | Description                                                              |
+|-----------------------|--------------------------------------------------------------------------|
+| `session_id`          | Session id of the blocked request                                        |
+| `blocking_session_id` | Session id of the direct blocker                                         |
+| `root_blocker`        | Session id at the head of the blocking chain                             |
+| `database`            | Database the blocked request runs in                                     |
+| `login`               | Login of the blocked session                                             |
+| `blocking_login`      | Login of the direct blocker                                              |
+| `wait_time`           | Seconds the request has been blocked (accepts units, e.g. `wait_time > 5m`) |
+| `wait_type`           | Wait type of the blocked request, e.g. `LCK_M_X`                         |
+| `command`             | Command the blocked request is executing, e.g. `UPDATE`                  |
+| `blocker_idle`        | `1` if the direct blocker has **no active request**                      |
+
+Defaults: **WARNING** on `wait_time > 30` (blocking that is already
+user-visible), **CRITICAL** on `wait_time > 300` (the application is frozen).
+Momentary lock waits below the thresholds are still counted and listed in the
+summary but do not alert. empty-state is **OK**: no blocked sessions is the
+healthy case.
+
+`root_blocker` resolves chains: when session 70 waits on 60 and 60 waits on
+50, both rows report `root_blocker = 50` — kill or investigate that session
+to release the whole chain. `blocker_idle = 1` identifies the classic
+orphaned-transaction case: the blocker is sleeping while holding locks inside
+an open transaction (an application that crashed or forgot to commit), which
+never resolves by itself — e.g.
+`warning=wait_time > 30s and blocker_idle = 1`.
+
+This is a point-in-time check: deadlocks are resolved by the engine within
+seconds and are therefore unlikely to be caught here — use the deadlock rate
+of `check_mssql_counters` for that. Sustained blocking, which this check is
+for, is exactly what deadlock detection does **not** resolve.
+
+Rights: `VIEW SERVER STATE`.

docs/samples/CheckMSSQL_check_mssql_blocking_samples.md

@@ -0,0 +1,34 @@
+**Default check (healthy — no blocking):**
+
+```
+check_mssql_blocking
+OK: No blocked sessions
+```
+
+**Default check during a blocking incident (warning at 30s, critical at 5m):**
+
+```
+check_mssql_blocking
+CRITICAL: 2/2 blocked sessions (appdb/app blocked by session 59 (app) for 506s on LCK_M_X, appdb/app blocked by session 60 (app) for 506s on LCK_M_X)|'60_wait_time'=506s;30;300 '61_wait_time'=506s;30;300
+```
+
+**Show the whole chain with the root blocker (the session to investigate):**
+
+```
+check_mssql_blocking "warning=none" "critical=wait_time > 30m" "top-syntax=${status}: ${list}" "detail-syntax=session ${session_id} (${login}) blocked by ${blocking_session_id} (${blocking_login}), root blocker ${root_blocker}, ${wait_time}s on ${wait_type}"
+OK: session 60 (app) blocked by 59 (app), root blocker 59, 506s on LCK_M_X, session 61 (app) blocked by 60 (app), root blocker 59, 506s on LCK_M_X|'60_wait_time'=506s;0;1800 '61_wait_time'=506s;0;1800
+```
+
+Here sessions 60 and 61 are both ultimately waiting on session 59 — killing or
+committing that one session releases the whole chain.
+
+**Alert only on orphaned transactions (idle blocker holding locks):**
+
+```
+check_mssql_blocking "warning=wait_time > 30s and blocker_idle = 1" "critical=wait_time > 30m"
+OK: 2 blocked sessions, none over the thresholds|'60_wait_time'=506s;30;1800 '61_wait_time'=506s;30;1800
+```
+
+The blocker in this incident was still actively executing (`blocker_idle = 0`),
+so the orphaned-transaction warning correctly stays quiet while the generic
+`wait_time` critical would still fire at 30 minutes.

docs/samples/CheckMSSQL_check_mssql_counters_desc.md

@@ -0,0 +1,44 @@
+#### About `check_mssql_counters`
+
+`check_mssql_counters` reports the **engine performance counters** every SQL
+Server health methodology starts with, from `sys.dm_os_performance_counters`
+over the same ODBC connection as every other check — so it works for remote
+and named instances where the local PDH counter sets
+(`SQLServer:Buffer Manager` etc.) are unavailable or renamed.
+
+Most of these counters are cumulative since instance start, so the check takes
+**two snapshots one second apart** (a server-side `WAITFOR DELAY`, the check
+takes ~1s longer than the others) and reports per-second rates over that
+window. The buffer cache hit ratio is likewise computed over the window — the
+lifetime ratio converges to ~100% on any long-running instance and hides a
+cold or thrashing cache.
+
+Keywords (one row per instance):
+
+| Keyword                | Description                                                            |
+|------------------------|------------------------------------------------------------------------|
+| `hit_ratio`            | Buffer cache hit ratio in percent over the sampling window             |
+| `page_life_expectancy` | Seconds a page stays in the buffer pool without being referenced       |
+| `batch_requests`       | Batch requests per second (the instance's workload pulse)              |
+| `compilations`         | SQL compilations per second (high vs `batch_requests` = plan-cache misuse) |
+| `recompilations`       | SQL re-compilations per second                                         |
+| `lazy_writes`          | Lazy-writer pages flushed per second; sustained values mean memory pressure |
+| `lock_waits`           | Lock requests per second that had to wait                              |
+| `deadlocks`            | Deadlocks per second across all lock types                             |
+
+Any counter reads `-1` when unavailable on the instance.
+
+All counters are emitted as **perfdata by default** (no threshold needed) —
+this check is primarily a graphing source. There are **no default alert
+thresholds** because healthy values scale with hardware and workload: page
+life expectancy scales with buffer pool size (the old "300 seconds" rule
+predates large-RAM servers; a common modern rule is 300s per 4 GB of buffer
+pool), and batch rates are only meaningful against your own baseline. Typical
+starting points:
+
+```
+check_mssql_counters "warning=hit_ratio < 95 or page_life_expectancy < 300" "critical=hit_ratio < 85 or page_life_expectancy < 60"
+check_mssql_counters "warning=deadlocks > 0.1" "critical=lazy_writes > 20"
+```
+
+Rights: `VIEW SERVER STATE`.

docs/samples/CheckMSSQL_check_mssql_counters_samples.md

@@ -0,0 +1,30 @@
+**Default check (informational, all counters emitted as perfdata):**
+
+```
+check_mssql_counters
+OK: hit ratio 100%, PLE 4010s, 155.819 batches/s, 0 compilations/s, 0 lazy writes/s, 0 lock waits/s, 0 deadlocks/s|'mssql_batch_requests'=155.81854;0;0 'mssql_compilations'=0;0;0 'mssql_deadlocks'=0;0;0 'mssql_hit_ratio'=100%;0;0 'mssql_lazy_writes'=0;0;0 'mssql_lock_waits'=0;0;0 'mssql_page_life_expectancy'=4010s;0;0 'mssql_recompilations'=0;0;0
+```
+
+The check samples the cumulative counters twice, one second apart, so it takes
+about a second longer than the other CheckMSSQL commands.
+
+**Alert on memory-pressure symptoms:**
+
+```
+check_mssql_counters "warning=hit_ratio < 95 or page_life_expectancy < 300" "critical=hit_ratio < 85 or page_life_expectancy < 60"
+OK: hit ratio 100%, PLE 4012s, 148.368 batches/s, 0 compilations/s, 0 lazy writes/s, 0 lock waits/s, 0 deadlocks/s|'mssql_batch_requests'=148.36795;0;0 'mssql_compilations'=0;0;0 'mssql_deadlocks'=0;0;0 'mssql_hit_ratio'=100%;95;85 'mssql_lazy_writes'=0;0;0 'mssql_lock_waits'=0;0;0 'mssql_page_life_expectancy'=4012s;300;60 'mssql_recompilations'=0;0;0
+```
+
+**A workload spike trips a batch-rate threshold:**
+
+```
+check_mssql_counters "warning=batch_requests > 100" "critical=batch_requests > 10000"
+WARNING: hit ratio 100%, PLE 4061s, 154.303 batches/s, 0 compilations/s, 0 lazy writes/s, 0 lock waits/s, 0 deadlocks/s|'mssql_batch_requests'=154.30267;100;10000 'mssql_compilations'=0;0;0 'mssql_deadlocks'=0;0;0 'mssql_hit_ratio'=100%;0;0 'mssql_lazy_writes'=0;0;0 'mssql_lock_waits'=0;0;0 'mssql_page_life_expectancy'=4061s;0;0 'mssql_recompilations'=0;0;0
+```
+
+**Watch locking health (pairs with `check_mssql_blocking`):**
+
+```
+check_mssql_counters "warning=lock_waits > 50" "critical=deadlocks > 0.5"
+OK: hit ratio 100%, PLE 4102s, 0 batches/s, 0 compilations/s, 0 lazy writes/s, 0 lock waits/s, 0 deadlocks/s|'mssql_batch_requests'=0;0;0 'mssql_compilations'=0;0;0 'mssql_deadlocks'=0;0;0.5 'mssql_hit_ratio'=100%;0;0 'mssql_lazy_writes'=0;0;0 'mssql_lock_waits'=0;50;0 'mssql_page_life_expectancy'=4102s;0;0 'mssql_recompilations'=0;0;0
+```

docs/samples/CheckMSSQL_check_mssql_databases_desc.md

@@ -16,6 +16,8 @@ Keywords (one row per database):
 | `data_size`      | Total data-file size in bytes (accepts units: `data_size > 10G`)                |
 | `log_size`       | Total log-file size in bytes (accepts units)                                    |
 | `log_used_pct`   | Percentage of the log in use, `-1` if unavailable                               |
+| `data_headroom`  | Smallest remaining growth room among the data files in bytes, `-1` if unavailable (accepts units: `data_headroom < 5G`) |
+| `log_headroom`   | Same for the log files (accepts units)                                          |
 
 Defaults: **CRITICAL** on broken states
 (`state = 'SUSPECT' or state = 'EMERGENCY' or state = 'RECOVERY_PENDING'`),
@@ -29,3 +31,19 @@ always exist, so an empty result indicates a broken query).
 permission for it (requires `VIEW SERVER STATE`), the check still works and
 reports `-1`. Perfdata is emitted for the size keywords referenced in your
 warning/critical expressions.
+
+`data_headroom`/`log_headroom` answer "how much further can this database
+grow before it errors": per file, the distance to `max_size` for capped files
+(the log's default 2TB cap counts as one), the free space on the file's
+volume (`sys.dm_os_volume_stats`) for uncapped autogrow files, and `0` when
+autogrowth is disabled — then the **minimum across the files of each type**,
+because SQL Server cannot move allocations between files and the most
+constrained file errors first. A threshold like
+`"critical=data_headroom < 1G and data_headroom >= 0"` catches both a file
+approaching its cap and a volume filling up — the `>= 0` guard excludes the
+`-1` unknown sentinel (unlike the plain size keywords, the headroom keywords
+accept plain integers as well as units, so `= -1` and `>= 0` work). Note that a fixed-size
+pre-allocated file reports headroom `0` by design — free space *inside* the
+files is a different measure (`log_used_pct` covers it for logs). Like
+`log_used_pct`, the keywords degrade to `-1` when `dm_os_volume_stats` is
+unavailable.

docs/samples/CheckMSSQL_check_mssql_databases_samples.md

@@ -32,3 +32,29 @@ OK: All 2 databases are ONLINE|'appdb_log_used_pct'=6%;80;90 'model_log_used_pct
 check_mssql_databases "filter=name != 'archive2019'"
 OK: All 5 databases are ONLINE
 ```
+
+**Alert before a file hits its max_size or fills its volume (`>= 0` excludes
+the `-1` unknown sentinel):**
+
+```
+check_mssql_databases "warning=data_headroom < 1K and data_headroom >= 0" "critical=log_headroom < 1K and log_headroom >= 0"
+OK: All 5 databases are ONLINE|'appdb_data_headroom'=96468992B;1024;0 'appdb_log_headroom'=2199014866944B;0;1024 'master_data_headroom'=991358509056B;1024;0 'master_log_headroom'=991358509056B;0;1024 'model_data_headroom'=991358509056B;1024;0 'model_log_headroom'=991358509056B;0;1024 'msdb_data_headroom'=991358509056B;1024;0 'msdb_log_headroom'=2199021944832B;0;1024 'tempdb_data_headroom'=991358509056B;1024;0 'tempdb_log_headroom'=991358509056B;0;1024
+```
+
+Uncapped autogrow files report the volume's free space; files capped by
+`max_size` report the distance to the cap (`appdb` here is capped at 100MB
+with ~92MB of room; the ~2TB log values are the engine's default log cap).
+
+**A database approaching its size cap trips the threshold:**
+
+```
+check_mssql_databases "warning=data_headroom < 200M and data_headroom >= 0" "critical=none" "top-syntax=${status}: ${list}" "detail-syntax=${name}: data headroom ${data_headroom}B, log headroom ${log_headroom}B"
+WARNING: appdb: data headroom 96468992B, log headroom 2199014866944B, master: data headroom 991358509056B, log headroom 991358509056B, model: data headroom 991358509056B, log headroom 991358509056B, msdb: data headroom 991358509056B, log headroom 2199021944832B, tempdb: data headroom 991358509056B, log headroom 991358509056B|'appdb_data_headroom'=96468992B;209715200;0 'master_data_headroom'=991358509056B;209715200;0 'model_data_headroom'=991358509056B;209715200;0 'msdb_data_headroom'=991358509056B;209715200;0 'tempdb_data_headroom'=991358509056B;209715200;0
+```
+
+**Flag hosts where headroom cannot be determined at all:**
+
+```
+check_mssql_databases "warning=data_headroom = -1" "critical=none"
+OK: All 5 databases are ONLINE|'appdb_data_headroom'=96468992B;-1;0 'master_data_headroom'=991358509056B;-1;0 'model_data_headroom'=991358509056B;-1;0 'msdb_data_headroom'=991358509056B;-1;0 'tempdb_data_headroom'=991358509056B;-1;0
+```

docs/samples/CheckMSSQL_check_mssql_integrity_desc.md

@@ -0,0 +1,32 @@
+#### About `check_mssql_integrity`
+
+`check_mssql_integrity` closes the gap `check_mssql_backup` leaves open: **a
+backup of a corrupt database restores a corrupt database.** It reports, per
+online database (tempdb excluded):
+
+| Keyword         | Description                                                                       |
+|-----------------|------------------------------------------------------------------------------------|
+| `name`          | Database name                                                                      |
+| `suspect_pages` | Pages in `msdb.dbo.suspect_pages` with unresolved 823/824/825 errors — any value above 0 means the engine has already **seen** corruption |
+| `checkdb_age`   | Seconds since the last successful `DBCC CHECKDB` (`dbi_dbccLastKnownGood`), `-1` = never checked, `-2` = unknown/no access (accepts units, e.g. `checkdb_age > 14d`) |
+
+Defaults: **CRITICAL** on `suspect_pages > 0` (corruption has occurred — act
+now, while the backups that can repair it still exist), **WARNING** on
+`checkdb_age > 14d or checkdb_age = -1` (corruption *would go unnoticed*).
+Restored and repaired pages (event types 4/5/7) are excluded from the count,
+so the alert clears once the damage is fixed.
+
+`checkdb_age = -2` means the CHECKDB timestamp could not be read and is
+deliberately quiet by default — reading it uses `DBCC DBINFO`, which requires
+**sysadmin**. The `suspect_pages` half works with `SELECT` on
+`msdb.dbo.suspect_pages` alone, so a low-privilege monitoring login still
+catches active corruption; add `warning=checkdb_age = -2` if you want missing
+access itself flagged. Ages are computed against the **server's own clock**,
+so an agent in a different timezone does not skew them.
+
+Note that `DBCC CHECKDB` itself is a heavy operation this check deliberately
+never runs — it only reads the timestamp the last run left behind. Schedule
+CHECKDB as a maintenance job (see `check_mssql_jobs` to alert when that job
+fails or stops running).
+
+Rights: `SELECT` on `msdb.dbo.suspect_pages`; `sysadmin` for `checkdb_age`.