Update documentation and add new diagnostic scripts for cache hit rates and job errors

Author: gmartinez-dbai

CONTRIBUTING.md

@@ -1,30 +1,28 @@
 # Contributing to pgtools
 
-We welcome contributions from the PostgreSQL community! This document provides comprehensive guidelines for contributing to the pgtools project.
+We welcome contributions from the PostgreSQL and TimescaleDB communities! This document provides guidelines for contributing to the `pgtools` "First Responder" Toolbelt.
 
-## 📋 Quick Start
+The goal of this project is to create a safe, reliable, and easy-to-use set of diagnostic scripts for support engineers and DBAs to use when triaging production database issues.
 
-### Prerequisites
-- PostgreSQL knowledge (administration, performance tuning, or development)
-- Basic understanding of SQL and shell scripting
-- Familiarity with Git and GitHub workflows
-- Access to PostgreSQL test environment for script validation
+## Core Principles
+
+All contributions must adhere to a strict **"Zero-Harm"** policy.
+
+1.  **Read-Only**: Scripts must *never* perform write operations. No `CREATE TABLE`, `ALTER`, `UPDATE`, `DELETE`, or `TRUNCATE`. Temporary tables should be avoided unless absolutely necessary and lightweight.
+2.  **Lightweight**: Queries must be efficient and avoid expensive operations that could impact a heavily loaded customer system.
+3.  **No Heavy Dependencies**: We stick to `bash` and `psql`. No Python, Go, or other languages that require complex installation.
+4.  **Safety First**: All queries are executed via the `pgtools.sh` wrapper, which enforces a short `statement_timeout` and `lock_timeout`.
+
+## Getting Started
 
-### Getting Started
 ```bash
 # Fork the repository on GitHub
 # Clone your fork
 git clone https://github.com/your-username/pgtools.git
 cd pgtools
 
-# Create development branch
+# Create a feature branch
 git checkout -b feature/your-feature-name
-
-# Test current scripts in your environment
-./automation/test_pgtools.sh --database your_test_db
-
-# Optional: run the full local pre-commit bundle
-./scripts/precommit_checks.sh --database your_test_db
 ```
 
 ## Types of Contributions

GETTING-STARTED.md

@@ -1,46 +1,29 @@
-# Getting Started with pgtools
+# Getting Started with pgtools: The First Responder Toolbelt
 
-Welcome to **pgtools** - the comprehensive PostgreSQL administration toolkit! This guide will help you get up and running quickly, whether you're a database administrator, developer, or DevOps engineer working with PostgreSQL.
+Welcome to **pgtools**, the "First Responder" Support Toolbelt for safely diagnosing PostgreSQL and TimescaleDB databases. This guide will help you get up and running quickly.
 
 ## 📋 Table of Contents
 
 - [Prerequisites](#prerequisites)
-- [Quick Installation](#quick-installation)
-- [First Steps](#first-steps)
-- [Essential Scripts](#essential-scripts)
-- [Common Workflows](#common-workflows)
-- [Setting Up Automation](#setting-up-automation)
-- [Troubleshooting](#troubleshooting)
-- [Next Steps](#next-steps)
+- [Installation](#installation)
+- [Core Usage](#core-usage)
+- [Example Workflows](#example-workflows)
+- [Available Commands](#available-commands)
 
 ## Prerequisites
 
-### System Requirements
-- **PostgreSQL**: Version 14 or higher (tested against 14, 15, 16, 17, and 18)
-- **Operating System**: Linux, macOS, or Windows with appropriate shell environment
-- **Shell**: Bash, Zsh, or compatible shell for automation scripts
-- **Tools**: `psql` command-line client, Git (for installation)
+- **Shell**: A `bash`-compatible shell.
+- **Tools**: `psql` (the PostgreSQL command-line client) and `git`.
+- **Database Access**: A valid PostgreSQL connection string or service name to connect to the target database. Most scripts require privileges equivalent to the `pg_monitor` role.
 
-### Database Access
-- **Privileges**: Most scripts require `pg_monitor` role or superuser privileges
-- **Connection**: Ability to connect to your PostgreSQL database(s)
-- **Extensions**: Some scripts benefit from `pg_stat_statements` and `pg_buffercache`
-
-### Knowledge Level
-- **Basic SQL**: Understanding of PostgreSQL queries and administration
-- **Command Line**: Comfort with terminal/command prompt usage
-- **PostgreSQL Concepts**: Familiarity with databases, tables, indexes, and basic administration
-
-## Quick Installation
-
-### Method 1: Git Clone (Recommended)
+## Installation
 ```bash
 # Clone the repository
 git clone https://github.com/thepostgresguy/pgtools.git
 cd pgtools
 
-# Make scripts executable
-chmod +x automation/*.sh maintenance/*.sh integration/*.sh configuration/*.sh
+# Make the main script executable
+chmod +x pgtools.sh
 ```
 
 ### Method 2: Download ZIP

cache-hit.sql

@@ -0,0 +1,51 @@
+/*
+ * Script: sql/diagnostics/cache-hit.sql
+ *
+ * Purpose: Calculates and displays table and index cache hit rates.
+ *
+ * Description:
+ * This query provides a crucial performance metric: the percentage of blocks
+ * read from the PostgreSQL buffer cache versus from disk. High cache hit
+ * rates (typically > 90-95%) indicate efficient memory usage and reduced
+ * reliance on slower disk I/O.
+ *
+ * Red Flags:
+ * - `table_hit_rate_pct` or `index_hit_rate_pct` consistently below 90-95%:
+ *   Indicates significant I/O pressure, potentially due to insufficient
+ *   `shared_buffers`, inefficient queries, or missing/bad indexes.
+ * - `total_reads` is very high for a table/index with a low hit rate:
+ *   This object is frequently accessed but rarely found in cache.
+ *
+ * Interpretation:
+ * - `table_hit_rate_pct`: Percentage of table blocks found in cache.
+ * - `index_hit_rate_pct`: Percentage of index blocks found in cache.
+ * - `total_reads`: Total number of blocks read (from cache + disk).
+ * - `total_hits`: Total number of blocks found in cache.
+ *
+ * Safety:
+ * This script is read-only. It queries `pg_stat_user_tables` and
+ * `pg_stat_user_indexes`, which are standard PostgreSQL statistics views
+ * designed for efficient diagnostic use. The `statement_timeout` set by
+ * `pgtools.sh` provides a safety guarantee.
+ */
+SELECT
+    relname AS object_name,
+    CASE WHEN relkind = 'r' THEN 'TABLE' WHEN relkind = 'i' THEN 'INDEX' END AS object_type,
+    blks_read + blks_hit AS total_reads,
+    blks_hit AS total_hits,
+    ROUND((blks_hit * 100.0 / NULLIF(blks_read + blks_hit, 0)), 2) AS hit_rate_pct
+FROM pg_stat_all_tables
+WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
+  AND (blks_read + blks_hit) > 0 -- Only show objects that have been accessed
+UNION ALL
+SELECT
+    relname AS object_name,
+    'INDEX' AS object_type,
+    idx_blks_read + idx_blks_hit AS total_reads,
+    idx_blks_hit AS total_hits,
+    ROUND((idx_blks_hit * 100.0 / NULLIF(idx_blks_read + idx_blks_hit, 0)), 2) AS hit_rate_pct
+FROM pg_stat_all_indexes
+WHERE schemaname NOT IN ('pg_catalog', 'information_schema')
+  AND (idx_blks_read + idx_blks_hit) > 0 -- Only show indexes that have been accessed
+ORDER BY hit_rate_pct ASC, total_reads DESC
+LIMIT 50;

job-errors.sql

@@ -0,0 +1,34 @@
+/*
+ * Script: sql/timescale/job-errors.sql
+ *
+ * Purpose: Shows recent errors from TimescaleDB background jobs.
+ *
+ * Description:
+ * This query retrieves the most recent errors logged by the TimescaleDB
+ * background worker scheduler. It is the primary tool for debugging why a
+ * policy (compression, CAGG refresh, retention) is failing.
+ *
+ * Red Flags:
+ * - Any rows returned are a red flag.
+ * - Repetitive `err_message` for the same `job_id`: Indicates a persistent problem.
+ * - `sqlerrcode` other than '00000': Provides the specific SQL error code.
+ *
+ * Interpretation:
+ * - `proc_name`: The type of job that failed (e.g., 'policy_compression').
+ * - `err_message`: The detailed error message, often explaining the root cause
+ *   (e.g., permission denied, constraint violation).
+ *
+ * Safety:
+ * This script is read-only. It queries the `timescaledb_information.job_errors`
+ * view, which is designed for efficient diagnostic use.
+ */
+SELECT
+  job_id,
+  proc_name,
+  start_time,
+  finish_time,
+  sqlerrcode,
+  err_message
+FROM timescaledb_information.job_errors
+ORDER BY start_time DESC
+LIMIT 50;

ownership.sql

@@ -0,0 +1,44 @@
+/*
+ * Script: sql/admin/ownership.sql
+ *
+ * Purpose: Displays ownership information for tables, views, and materialized views.
+ *
+ * Description:
+ * This query lists all user-defined tables, views, materialized views, and
+ * foreign tables, along with their respective owners. It's a key diagnostic
+ * for "permission denied" errors or when auditing for orphaned objects after
+ * role changes.
+ *
+ * Red Flags:
+ * - Objects owned by roles that no longer exist (owner will show as an OID).
+ * - Critical application tables owned by a superuser or an unexpected role.
+ * - Inconsistent ownership patterns across schemas or applications.
+ *
+ * Interpretation:
+ * - `schema_name`: The schema containing the object.
+ * - `object_name`: The name of the table, view, etc.
+ * - `object_type`: Indicates if it's a TABLE, VIEW, MATERIALIZED VIEW, etc.
+ * - `owner`: The role that owns the object.
+ *
+ * Safety:
+ * This script is read-only. It queries standard `pg_catalog` views (`pg_class`,
+ * `pg_namespace`) which are designed for efficient diagnostic use. The
+ * `statement_timeout` set by `pgtools.sh` provides a safety guarantee.
+ */
+SELECT
+    n.nspname AS schema_name,
+    c.relname AS object_name,
+    pg_catalog.pg_get_userbyid(c.relowner) AS owner,
+    CASE c.relkind
+        WHEN 'r' THEN 'TABLE'
+        WHEN 'v' THEN 'VIEW'
+        WHEN 'm' THEN 'MATERIALIZED VIEW'
+        WHEN 'f' THEN 'FOREIGN TABLE'
+        WHEN 'p' THEN 'PARTITIONED TABLE'
+    END AS object_type
+FROM pg_catalog.pg_class c
+JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
+WHERE c.relkind IN ('r','v','m','f','p')
+    AND n.nspname NOT IN ('pg_catalog', 'information_schema')
+    AND n.nspname !~ '^pg_toast'
+ORDER BY n.nspname, c.relname;

pgtools.sh

@@ -67,11 +67,14 @@ Commands:
   bloat             Identify table and index bloat. (sql/diagnostics/bloat.sql)
   replication       Monitor replication lag and status. (sql/diagnostics/replication.sql)
   disk-usage        Show disk usage by table and index. (sql/diagnostics/disk-usage.sql)
+  cache-hit         Show table and index cache hit rates. (sql/diagnostics/cache-hit.sql)
 
   # TimescaleDB Diagnostics
   chunk-stats       Show chunk count and size per hypertable. (sql/timescale/chunk-stats.sql)
   compression-stats Show compression ratio and job status per hypertable. (sql/timescale/compression-stats.sql)
   cagg-stats        Show continuous aggregate health and refresh policy status. (sql/timescale/cagg-stats.sql)
+  job-errors        Show recent errors from background jobs. (sql/timescale/job-errors.sql)
+  uncompressed-chunks Show chunks that are old but not compressed. (sql/timescale/uncompressed-chunks.sql)
 
   # Administration
   permissions       Audit user and role permissions. (sql/admin/permissions.sql)
@@ -123,7 +126,7 @@ main() {
     local connection_string="$2"
 
     case "${command}" in
-        locks|activity|top-queries|bloat|replication|disk-usage|chunk-stats|compression-stats|cagg-stats|permissions|ownership)
+        locks|activity|top-queries|bloat|replication|disk-usage|cache-hit|chunk-stats|compression-stats|cagg-stats|job-errors|uncompressed-chunks|permissions|ownership)
             run_sql "${command}" "${connection_string}"
             ;;
         *)