Skip to content

Commit bd7af61

Browse files
Kevin NgoKevin Ngo
Kevin Ngo
authored and
Kevin Ngo
committed
add five observability and diagnostics skills
Add five new skills to the observability-and-diagnostics domain for CockroachDB performance analysis and troubleshooting: 1. analyzing-range-distribution - Analyzes range distribution and replica placement across cluster - Identifies hotspots and load imbalances using crdb_internal.ranges - Provides queries for range counts, size distribution, and replica balance 2. analyzing-schema-change-storage-risk - Assesses storage impact of schema changes before execution - Estimates space requirements for backfills and index builds - Uses crdb_internal tables for safe production analysis 3. auditing-table-statistics - Audits table statistics freshness and quality for query optimizer - Identifies stale or missing statistics affecting plan quality - Provides queries for statistics age, row count accuracy, and coverage 4. monitoring-background-jobs - Monitors long-running background jobs (schema changes, backups, imports) - Tracks job progress, status, and failure diagnostics - Uses crdb_internal.jobs for production-safe job monitoring 5. profiling-transaction-fingerprints - Analyzes transaction fingerprints using aggregated statistics - Identifies high-retry transactions, commit latency patterns - Provides statement composition drill-down via stmtFingerprintIDs - Includes 7 core diagnostic queries and 4 common workflows Also updates profiling-statement-fingerprints and triaging-live-sql-activity with cross-references to the new skills for complementary analysis workflows.
1 parent 0a9254e commit bd7af61

20 files changed

Lines changed: 8928 additions & 0 deletions

File tree

‎skills/observability-and-diagnostics/analyzing-range-distribution/SKILL.md‎

Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
---
2+
name: analyzing-range-distribution
3+
description: Analyzes CockroachDB range distribution across tables and indexes using SHOW RANGES to identify range count, size patterns, leaseholder placement, and replication health. Use when investigating hotspots, uneven data distribution, range fragmentation, or validating zone configuration effects without DB Console access.
4+
compatibility: Requires SQL access with admin role or ZONECONFIG system privilege. DETAILS option has high cost; use targeted queries with LIMIT. Production-safe for basic usage.
5+
metadata:
6+
author: cockroachdb
7+
version: "1.0"
8+
---
9+
10+
# Analyzing Range Distribution
11+
12+
Analyzes CockroachDB range distribution, leaseholder placement, and zone configuration compliance using `SHOW RANGES` and `SHOW ZONE CONFIGURATIONS` commands. Identifies range count anomalies, size imbalances, leaseholder hotspots, and replication issues - entirely via SQL without requiring DB Console access.
13+
14+
**Complement to profiling skills:** This skill analyzes range-level data distribution; for query performance patterns, see [profiling-statement-fingerprints](../profiling-statement-fingerprints/SKILL.md). For schema change storage planning, see [analyzing-schema-change-storage-risk](../analyzing-schema-change-storage-risk/SKILL.md).
15+
16+
## When to Use This Skill
17+
18+
- Identify tables/indexes with excessive range counts indicating fragmentation
19+
- Detect range size imbalances or uneven data distribution across nodes
20+
- Investigate leaseholder concentration causing read hotspots
21+
- Validate zone configuration effects on range placement and replica distribution
22+
- Diagnose range-level replication issues (under-replicated or unavailable ranges)
23+
- Analyze range split patterns from high write volume
24+
- SQL-only range analysis without DB Console access
25+
26+
**For schema change planning:** Use [analyzing-schema-change-storage-risk](../analyzing-schema-change-storage-risk/SKILL.md) to estimate storage requirements before CREATE INDEX or ADD COLUMN operations.
27+
28+
## Prerequisites
29+
30+
- SQL connection to CockroachDB cluster
31+
- Admin role OR `ZONECONFIG` system privilege
32+
- Understanding of CockroachDB range architecture (64MB default max size)
33+
- Knowledge of cluster topology (node IDs, regions, availability zones)
34+
35+
**Check your privileges:**
36+
```sql
37+
SHOW GRANTS ON SYSTEM FOR current_user; -- Should show admin or ZONECONFIG
38+
```
39+
40+
See [permissions reference](references/permissions.md) for RBAC setup.
41+
42+
## Core Concepts
43+
44+
### Ranges: Units of Data Distribution
45+
46+
**Range:** Contiguous key space segment (default 64MB max size, configurable via zone config `range_max_bytes`)
47+
**Raft group:** Each range replicated across nodes (default 3 replicas)
48+
**Leaseholder:** Single replica handling reads and coordinating writes for a range
49+
50+
**Critical:** Ranges split automatically at 64MB by default, but can fragment further due to load-based splitting during high write traffic.
51+
52+
### Leaseholders and Hotspots
53+
54+
**Leaseholder concentration:** Single node holding disproportionate leaseholders = read hotspot
55+
**Load-based splitting:** CockroachDB splits ranges experiencing high QPS, increasing range count
56+
**Hotspot symptoms:** High CPU on single node, slow reads on specific table/index
57+
58+
### Range Fragmentation
59+
60+
**Fragmentation:** Excessive range splits creating many small ranges (overhead from Raft coordination)
61+
**Causes:** High write throughput, sequential inserts (timestamp-based primary keys), load-based splitting
62+
**Symptoms:** High range count relative to data size, increased latency from Raft overhead
63+
64+
**Fragmentation metric:** Ranges per GB (healthy: 1-15, fragmented: 50+)
65+
66+
### Zone Configurations
67+
68+
**Zone config:** Replication and placement policies for databases, tables, or indexes
69+
**Replication factor:** Number of replicas per range (default: 3)
70+
**Constraints:** Node placement rules (region, availability zone, node attributes)
71+
72+
**Use case:** Validate intended zone config matches actual range placement.
73+
74+
### SHOW RANGES DETAILS Option
75+
76+
**CRITICAL SAFETY WARNING:** The `WITH DETAILS` option computes `span_stats` (range size, key counts) on-demand, causing:
77+
- **High CPU usage** from statistics computation
78+
- **Memory overhead** proportional to range count
79+
- **Query timeouts** on large tables without LIMIT
80+
81+
**Best practice:** Always use `LIMIT` with `DETAILS`, target specific tables/indexes, avoid cluster-wide scans.
82+
83+
## Core Diagnostic Queries
84+
85+
### Query 1: Range Count by Table (Production-Safe)
86+
87+
```sql
88+
SELECT
89+
table_name,
90+
index_name,
91+
COUNT(*) AS range_count
92+
FROM [SHOW RANGES FROM TABLE your_table_name]
93+
GROUP BY table_name, index_name
94+
ORDER BY range_count DESC;
95+
```
96+
97+
**Interpretation:** High range count (1000s) on small tables indicates fragmentation. Cross-reference with table size.
98+
99+
**Safety:** No `DETAILS` option = production-safe, minimal overhead.
100+
101+
### Query 2: Range Size Analysis (Targeted DETAILS)
102+
103+
```sql
104+
SELECT
105+
range_id,
106+
start_key,
107+
end_key,
108+
(span_stats->>'approximate_disk_bytes')::INT / 1048576 AS size_mb,
109+
lease_holder,
110+
replicas
111+
FROM [SHOW RANGES FROM TABLE your_table_name WITH DETAILS]
112+
ORDER BY (span_stats->>'approximate_disk_bytes')::INT DESC
113+
LIMIT 50;
114+
```
115+
116+
**Interpretation:** Large ranges (>64MB) indicate split lag; many small ranges (<10MB) indicate fragmentation.
117+
118+
**CRITICAL:** Always include `LIMIT` and target specific tables. Never run `SHOW RANGES WITH DETAILS` on entire database.
119+
120+
### Query 3: Leaseholder Distribution (Hotspot Detection)
121+
122+
```sql
123+
SELECT
124+
lease_holder,
125+
COUNT(*) AS leaseholder_count,
126+
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) AS percentage
127+
FROM [SHOW RANGES FROM TABLE your_table_name]
128+
GROUP BY lease_holder
129+
ORDER BY leaseholder_count DESC;
130+
```
131+
132+
**Interpretation:** >40% leaseholders on single node in balanced cluster = hotspot. Check if table has zone constraints favoring specific nodes.
133+
134+
**Remediation:** Use `ALTER TABLE ... CONFIGURE ZONE USING lease_preferences` to spread leaseholders.
135+
136+
### Query 4: Range Replication Health Check
137+
138+
```sql
139+
SELECT
140+
range_id,
141+
start_key,
142+
replicas,
143+
array_length(replicas, 1) AS replica_count,
144+
voting_replicas,
145+
array_length(voting_replicas, 1) AS voting_replica_count,
146+
lease_holder
147+
FROM [SHOW RANGES FROM TABLE your_table_name]
148+
WHERE array_length(replicas, 1) < 3 -- Under-replicated
149+
ORDER BY range_id
150+
LIMIT 100;
151+
```
152+
153+
**Interpretation:** `replica_count < 3` = under-replicated (data loss risk). Check for node failures, decommissioning operations, or zone config mismatches.
154+
155+
**Safety:** No `DETAILS` = production-safe.
156+
157+
### Query 5: Zone Configuration Audit
158+
159+
```sql
160+
SHOW ZONE CONFIGURATIONS;
161+
```
162+
163+
**Output columns:**
164+
- `target`: Database, table, or index
165+
- `raw_config_sql`: Zone config SQL (replication factor, constraints)
166+
167+
**Use case:** Validate intended replication factor and placement constraints match expected design.
168+
169+
**Cross-reference:** Compare zone configs with Query 3 (leaseholder distribution) and Query 4 (replica health) to validate actual placement.
170+
171+
### Query 6: Fragmentation Analysis (Ranges per GB)
172+
173+
```sql
174+
WITH range_counts AS (
175+
SELECT
176+
table_name,
177+
index_name,
178+
COUNT(*) AS range_count
179+
FROM [SHOW RANGES FROM TABLE your_table_name]
180+
GROUP BY table_name, index_name
181+
),
182+
table_sizes AS (
183+
SELECT
184+
table_name,
185+
SUM((span_stats->>'approximate_disk_bytes')::INT) / 1073741824.0 AS size_gb
186+
FROM [SHOW RANGES FROM TABLE your_table_name WITH DETAILS]
187+
GROUP BY table_name
188+
)
189+
SELECT
190+
rc.table_name,
191+
rc.index_name,
192+
rc.range_count,
193+
ts.size_gb,
194+
ROUND(rc.range_count / NULLIF(ts.size_gb, 0), 2) AS ranges_per_gb
195+
FROM range_counts rc
196+
JOIN table_sizes ts ON rc.table_name = ts.table_name
197+
ORDER BY ranges_per_gb DESC;
198+
```
199+
200+
**Interpretation:**
201+
- **Healthy:** 1-15 ranges/GB
202+
- **Moderate fragmentation:** 16-50 ranges/GB
203+
- **Severe fragmentation:** 50+ ranges/GB
204+
205+
**CRITICAL:** This query uses `DETAILS` - only run on targeted tables with known size, never cluster-wide.
206+
207+
**Remediation:** Increase `range_max_bytes` via zone config (with caution), or accept fragmentation if caused by necessary load-based splitting.
208+
209+
See [sql-queries reference](references/sql-queries.md) for complete query variations and guardrails.
210+
211+
## Common Workflows
212+
213+
### Workflow 1: Hotspot Investigation
214+
215+
**Scenario:** Single node experiencing high CPU, slow reads on specific table.
216+
217+
**Steps:**
218+
1. **Identify leaseholder concentration:** Run Query 3 on suspected table
219+
2. **Validate zone config:** Run Query 5 to check lease_preferences
220+
3. **Check for load-based splits:** Run Query 1 to detect recent range fragmentation (symptom of hotspot)
221+
4. **Remediate:** Configure lease preferences to spread reads, or partition table if hotspot is on sequential key range
222+
223+
**Example:**
224+
```sql
225+
-- Check leaseholder distribution
226+
SELECT lease_holder, COUNT(*) FROM [SHOW RANGES FROM TABLE hot_table] GROUP BY lease_holder;
227+
228+
-- Validate zone config
229+
SHOW ZONE CONFIGURATION FOR TABLE hot_table;
230+
231+
-- Spread leaseholders if concentrated
232+
ALTER TABLE hot_table CONFIGURE ZONE USING lease_preferences = '[[+region=us-west]]';
233+
```
234+
235+
### Workflow 2: Zone Config Validation
236+
237+
**Scenario:** After configuring multi-region setup, validate ranges are placed according to constraints.
238+
239+
**Steps:**
240+
1. **Review intended configs:** Run Query 5 (SHOW ZONE CONFIGURATIONS)
241+
2. **Check actual replica placement:** Run Query 4 on critical tables, inspect `replicas` array for node IDs
242+
3. **Map node IDs to regions:** Cross-reference with `SHOW REGIONS` or `crdb_internal.gossip_nodes`
243+
4. **Identify mismatches:** Ranges not matching constraints indicate rebalancing in progress or misconfiguration
244+
245+
**Example:**
246+
```sql
247+
-- Show zone config
248+
SHOW ZONE CONFIGURATION FOR TABLE multi_region_table;
249+
250+
-- Check replica placement
251+
SELECT range_id, replicas FROM [SHOW RANGES FROM TABLE multi_region_table] LIMIT 20;
252+
253+
-- Map node IDs to regions
254+
SELECT node_id, locality FROM crdb_internal.gossip_nodes;
255+
```
256+
257+
### Workflow 3: Fragmentation Diagnosis
258+
259+
**Scenario:** Table with high range count relative to size, experiencing latency.
260+
261+
**Steps:**
262+
1. **Calculate ranges per GB:** Run Query 6 (targeted to specific table)
263+
2. **Check for load-based splits:** Review write patterns (sequential inserts, high QPS periods)
264+
3. **Determine if expected:** Fragmentation may be intentional for load distribution
265+
4. **Remediate if excessive:** Increase `range_max_bytes` (with caution - larger ranges = slower splits), or investigate reducing write hotspots
266+
267+
**CRITICAL:** Never increase `range_max_bytes` above 512MB without understanding impact on split/rebalance performance.
268+
269+
## Safety Considerations
270+
271+
### DETAILS Option Cost
272+
273+
**Resource impact:**
274+
- **CPU:** Computes span statistics on-demand for each range
275+
- **Memory:** Proportional to range count returned
276+
- **Timeout risk:** High on tables with 1000s of ranges without LIMIT
277+
278+
**Mitigation strategies:**
279+
1. **Always use LIMIT:** Cap at 50-100 ranges for exploratory analysis
280+
2. **Target specific tables:** Use `FROM TABLE table_name`, never cluster-wide `SHOW RANGES WITH DETAILS`
281+
3. **Use basic queries first:** Run Query 1 (no DETAILS) to assess range count before using DETAILS
282+
4. **Production timing:** Run during maintenance windows or low-traffic periods
283+
284+
### Privilege Safety
285+
286+
**Admin role:** Full cluster access, use with caution in production
287+
**ZONECONFIG privilege:** Limited to viewing ranges and zone configs, safer for read-only analysis
288+
289+
**Best practice:** Grant `ZONECONFIG` instead of admin for range analysis operators.
290+
291+
See [permissions reference](references/permissions.md) for granting minimal privileges.
292+
293+
### Production Impact
294+
295+
**Read-only operations:** All queries are `SELECT` or `SHOW` statements with no writes.
296+
297+
**Performance considerations:**
298+
299+
| Query Type | Impact | Safe for Production? |
300+
|------------|--------|---------------------|
301+
| Basic SHOW RANGES | Minimal CPU, metadata-only | Yes |
302+
| SHOW RANGES WITH DETAILS (targeted, LIMIT 50) | Moderate CPU spike | Yes (low-traffic window) |
303+
| SHOW RANGES WITH DETAILS (no LIMIT) | High CPU, timeout risk | **NO - NEVER USE** |
304+
| SHOW ZONE CONFIGURATIONS | Minimal, metadata-only | Yes |
305+
306+
## Troubleshooting
307+
308+
| Issue | Cause | Fix |
309+
|-------|-------|-----|
310+
| Permission denied | Missing admin or ZONECONFIG privilege | Grant ZONECONFIG: `GRANT SYSTEM ZONECONFIG TO user` |
311+
| Query timeout with DETAILS | Too many ranges without LIMIT | Add `LIMIT 50`, target specific table |
312+
| Empty span_stats column | Missing DETAILS keyword | Add `WITH DETAILS` to SHOW RANGES |
313+
| Unexpected high range count | Load-based splitting or fragmentation | Run Query 6 to calculate ranges/GB, review write patterns |
314+
| Leaseholder = 0 or NULL | Range in transition during rebalancing | Normal during cluster changes, retry query |
315+
| Under-replicated ranges | Node failure, decommission, zone mismatch | Check node status, validate zone config constraints |
316+
| SHOW ZONE CONFIGURATIONS shows no custom configs | Using default cluster-wide config | Normal if no table/database-level overrides set |
317+
318+
## Key Considerations
319+
320+
- **DETAILS option:** Expensive operation - always use with LIMIT and targeted scope
321+
- **Fragmentation is sometimes intentional:** Load-based splitting improves concurrency
322+
- **Leaseholder concentration:** Check zone configs (lease_preferences) before assuming hotspot
323+
- **Range size target:** Default 64MB max (not 512MB as in older versions)
324+
- **Replication lag:** Range placement may not immediately reflect zone config changes (rebalancing takes time)
325+
- **Cross-reference queries:** Combine range analysis with zone configs for complete picture
326+
- **Node mapping:** Use `crdb_internal.gossip_nodes` to map node IDs to regions/zones
327+
328+
## References
329+
330+
**Skill references:**
331+
- [SQL query variations and guardrails](references/sql-queries.md)
332+
- [RBAC and privileges setup](references/permissions.md)
333+
334+
**Official CockroachDB Documentation:**
335+
- [SHOW RANGES](https://www.cockroachlabs.com/docs/stable/show-ranges.html)
336+
- [SHOW ZONE CONFIGURATIONS](https://www.cockroachlabs.com/docs/stable/show-zone-configurations.html)
337+
- [Architecture: Distribution Layer](https://www.cockroachlabs.com/docs/stable/architecture/distribution-layer.html)
338+
- [Configure Replication Zones](https://www.cockroachlabs.com/docs/stable/configure-replication-zones.html)
339+
- [ZONECONFIG privilege](https://www.cockroachlabs.com/docs/stable/security-reference/authorization.html#supported-privileges)
340+
341+
**Related skills:**
342+
- [profiling-statement-fingerprints](../profiling-statement-fingerprints/SKILL.md) - For query performance analysis
343+
- [triaging-live-sql-activity](../triaging-live-sql-activity/SKILL.md) - For real-time query triage
344+
- [analyzing-schema-change-storage-risk](../analyzing-schema-change-storage-risk/SKILL.md) - For estimating storage requirements before DDL operations

0 commit comments

Comments
 (0)