Refactor README to enhance clarity and organization of diagnostic scripts and usage examples

Author: gmartinez-dbai

README.md

@@ -1,288 +1,103 @@
-# pgtools
-
-A collection of SQL scripts and utilities for monitoring, troubleshooting, and maintaining PostgreSQL databases.
-
-## 👋 New to pgtools?
-
-**[👉 Get Started Here - Complete Beginner's Guide](GETTING-STARTED.md)**
-
-Perfect for new users! This comprehensive guide walks you through installation, first steps, essential workflows, and automation setup.
-
-## 📋 Table of Contents
-
-- [Overview](#overview)
-- [Script Categories](#script-categories)
-- [Usage Examples](#usage-examples)
-- [Contributing](#contributing)
-- [License](#license)
-
-### Quick Links
-
-This toolkit provides battle-tested SQL scripts for PostgreSQL database administrators and developers to:
-- Monitor database health and performance
-- Troubleshoot common issues
-- Maintain database integrity
-- Optimize query performance
-- Manage replication and WAL files
-## Script Categories
-### 🔍 Monitoring Scripts
-**bloating.sql**
-- Detects table and index bloat
-- Shows dead tuples and wasted space
-- Helps identify tables needing VACUUM
-
-**buffer_troubleshoot.sql**
-- Analyzes shared buffer usage
-- Shows buffer cache hit ratios
-- Identifies tables with poor caching
-
-**locks.sql**
-- Lists current locks in the database
-- Shows lock types and waiting queries
-- Essential for deadlock investigation
-
-**postgres_locking_blocking.sql**
-- Advanced lock analysis
-- Shows blocking and blocked queries
-- Includes query details and wait times
-
-**replication.sql**
-- Monitors replication lag
-- Shows replication slot status
-- Checks standby server health
-
-**txid.sql**
-- Displays transaction ID information
-- Monitors transaction wraparound risk
-- Shows age of databases and tables
-
-**connection_pools.sql**
-- Monitors connection pooling health and efficiency
-- Analyzes connection patterns and potential leaks
-- Provides connection pool optimization recommendations
-- Works with PgBouncer, Pgpool-II, and native connections
-
-### 🔧 Maintenance Scripts
-**switch_pg_wal_file.sql**
-- Forces WAL file switching
-- Useful for archiving and backup operations
-- Requires superuser privileges
-
-**walfile_in_use.sql**
-- Shows currently active WAL files
-- Displays WAL file location and size
-- Helps troubleshoot disk space issues
-
-**Transaction Wraparound**
-- Scripts for monitoring and preventing transaction ID wraparound
-- Critical for database availability
-
-### 🤖 Maintenance Automation
-**auto_maintenance.sh**
-- Comprehensive automated maintenance operations (VACUUM, ANALYZE, REINDEX)
-- Intelligent threshold-based maintenance with configurable parameters
-- Parallel processing with safety controls and dry-run mode
-- Large table detection and resource management
-
-**maintenance_scheduler.sql**
-- Analysis and scheduling recommendations for maintenance operations
-- VACUUM/ANALYZE candidate identification with workload estimation
-- Index bloat analysis and autovacuum effectiveness assessment
-- Maintenance planning and resource optimization
-
-**statistics_collector.sql**
-- Table and index statistics analysis and optimization
-- Statistics quality assessment and freshness analysis
-- Column distribution analysis with optimization recommendations
-- Extended statistics support for PostgreSQL 15+
-
-### 👤 Administration Scripts
-**extensions.sql**
-- Lists installed PostgreSQL extensions
-- Shows extension versions and schemas
-- Helps audit database capabilities
-
-**table_ownership.sql**
-- Shows table ownership information
-- Useful for permission audits
-- Helps with database migrations
-
-**ForeignConst.sql**
-- Lists foreign key constraints
-- Shows constraint details and relationships
-- Aids in schema documentation
-
-**NonHypertables.sql**
-- Identifies non-hypertables (TimescaleDB specific)
-- Useful for TimescaleDB users
-- Helps in migration planning
-
-**partition_management.sql**
-- Monitors partition health and performance
-- Analyzes partition size distribution and balance
-- Provides partition maintenance recommendations
-- Supports automated partition management strategies
-
-### ⚡ Optimization Scripts
-**hot_update_optimization_checklist.sql**
-- Checks HOT (Heap-Only Tuple) update optimization
-- Identifies inefficient table structures
-- Suggests fillfactor adjustments
-
-**missing_indexes.sql**
-- Identifies potentially beneficial indexes based on query patterns
-- Analyzes sequential scan activity and unused indexes
-- Detects foreign key columns missing indexes
-- Provides index optimization recommendations
-
-### 📦 Backup & Recovery Scripts
-**backup_validation.sql**
-- Validates backup completeness and integrity
-- Checks WAL archiving status and health
-- Analyzes backup readiness and configuration
-- Provides backup strategy recommendations
-
-### 🔒 Security Scripts
-**permission_audit.sql**
-- Comprehensive security audit of roles and permissions
-- Identifies overprivileged accounts and security risks
-- Analyzes database, schema, and table-level access
-- Reviews authentication and Row Level Security (RLS)
-
-### ⚡ Performance Analysis Scripts
-**wait_event_analysis.sql**
-- Comprehensive analysis of PostgreSQL wait events and performance bottlenecks
-- Identifies I/O, locking, and resource contention issues
-- Provides detailed wait event categorization and recommendations
-- Analyzes connection pooling and background worker efficiency
-
-**query_performance_profiler.sql**
-- Detailed query performance analysis using pg_stat_statements
-- Identifies slow queries, I/O intensive operations, and resource usage
-- Analyzes query variance and performance degradation patterns
-- Provides optimization recommendations for query tuning
-
-**resource_monitoring.sql**
-- Comprehensive system resource utilization monitoring
-- Analyzes memory, I/O, connection, and storage usage patterns
-- Monitors autovacuum activity and maintenance requirements
-- Provides resource optimization recommendations
-
-### ⚙️ Configuration Management Scripts
-**configuration_analysis.sql**
-- Comprehensive PostgreSQL configuration analysis and recommendations
-- Reviews memory, connection, WAL, and security settings
-- Analyzes current parameters against best practices
-- Provides workload-specific tuning suggestions
-
-**parameter_tuner.sh** (automation/configuration/)
-- Interactive PostgreSQL parameter tuning assistant
-- Generates optimized configurations for different workload types (OLTP, OLAP, Web)
-- Provides memory and performance setting recommendations
-- Supports configuration validation and analysis modes
-
-### 🔗 Integration Tools
-**grafana_dashboard_generator.sh** (integration/)
-- Generates comprehensive Grafana dashboards for PostgreSQL monitoring
-- Supports multiple dashboard types: comprehensive, performance, security, connections
-- Provides direct Grafana API integration for automatic dashboard deployment
-- Creates customizable monitoring visualizations
-
-**prometheus_exporter.sh** (integration/)
-- Custom PostgreSQL metrics exporter for Prometheus
-- Exports database statistics, connection metrics, and performance data
-- Supports daemon mode for continuous metrics collection
-- Provides HTTP endpoint for Prometheus scraping
-
-### 🩺 Troubleshooting Scripts
-**postgres_troubleshooting_queries.sql**
-- Collection of diagnostic queries
-- Quick health checks
-- Performance analysis queries
-
-**postgres_troubleshooting_query_pack_01.sql**
-- First pack of troubleshooting queries
-- Focuses on basic diagnostics
-
-**postgres_troubleshooting_query_pack_02.sql**
-- Second pack of troubleshooting queries
-- Intermediate level diagnostics
-
-**postgres_troubleshooting_query_pack_03.sql**
-- Third pack of troubleshooting queries
-- Advanced diagnostics
-
-**postgres_troubleshooting_cheat_sheet.txt**
-- Quick reference guide
-- Common commands and queries
-- Best practices and tips
-
-## Usage Examples
-### Check for blocking queries
-```bash
-psql -U postgres -d mydb -f monitoring/postgres_locking_blocking.sql
-```
-### Monitor replication lag
-```bash
-psql -U postgres -d mydb -f monitoring/replication.sql
-```
-### Identify bloated tables
+# pgtools: The First Responder's Toolbelt for PostgreSQL
+
+`pgtools` is a curated collection of safe, read-only diagnostic scripts for PostgreSQL and TimescaleDB, wrapped in a simple command-line interface. It is designed for Support Engineers, DBAs, and developers who need to triage production database issues quickly and without causing harm.
+
+Every script is executed with strict, short timeouts to ensure that diagnostic queries never impact a heavily loaded system.
+
+## Core Principles
+
+- **Zero-Harm Policy**: Every script is read-only and executed with a 5-second `statement_timeout` and 3-second `lock_timeout`.
+- **No Dependencies**: The toolbelt relies only on `bash` and `psql`. No Python, Go, or other complex dependencies are required.
+- **Ticket-Ready Output**: All output is formatted for easy copy-pasting into Zendesk, Jira, or Markdown documents.
+- **Community-Driven**: Built for general PostgreSQL users, with specialized diagnostics for TimescaleDB.
+
+## Getting Started
+
+### Installation
+
 ```bash
-psql -U postgres -d mydb -f monitoring/bloating.sql
+# 1. Clone the repository
+git clone <https://github.com/thepostgresguy/pgtools.git>
+cd pgtools
+
+# 2. Make the wrapper script executable
+chmod +x pgtools.sh
 ```
-### Check transaction wraparound risk
+
+### Usage
+
+All commands are run through the `pgtools.sh` wrapper.
+
 ```bash
-psql -U postgres -d mydb -f monitoring/txid.sql
+./pgtools.sh <command> "<connection_string>"
 ```
-### Validate backup readiness
+
+**Example: Check for blocking locks**
 ```bash
-psql -U postgres -d mydb -f backup/backup_validation.sql
+./pgtools.sh locks "postgresql://user:pass@host:port/dbname"
 ```
-### Analyze connection pooling efficiency
+
+**Example: Check TimescaleDB chunk stats using a service name**
 ```bash
-psql -U postgres -d mydb -f monitoring/connection_pools.sql
+./pgtools.sh chunk-stats "service=my_customer_db"
 ```
 
-### Automation / HOT report verification
-```bash
-# Quick automation sanity check (connection, syntax, permissions)
-./automation/test_pgtools.sh --fast
+## Available Commands
 
-# Full automation suite with integration tests
-./automation/test_pgtools.sh --full --verbose
+Run `./pgtools.sh` with no arguments to see the full list of commands.
 
-# HOT checklist JSON validation
-./automation/run_hot_update_report.sh --format json --database my_database --stdout
+### General Diagnostics
 
-# HOT checklist text validation
-./automation/run_hot_update_report.sh --format text --database my_database --stdout
+*   `locks`: Show current lock contention and blocking queries.
+*   `activity`: Display current query activity from `pg_stat_activity`.
+*   `top-queries`: Show most time-consuming queries (requires `pg_stat_statements`).
+*   `bloat`: Identify table and index bloat.
+*   `replication`: Monitor replication lag and status.
+*   `disk-usage`: Show disk usage by table and index.
+*   `cache-hit`: Show table and index cache hit rates.
 
-# Full local pre-commit bundle
-./scripts/precommit_checks.sh --database my_database
-```
+### TimescaleDB Diagnostics
+
+*   `chunk-stats`: Show chunk count and size per hypertable.
+*   `compression-stats`: Show compression ratio and job status per hypertable.
+*   `cagg-stats`: Show continuous aggregate health and refresh policy status.
+*   `job-errors`: Show recent errors from background jobs.
+*   `uncompressed-chunks`: Show chunks that are old but not compressed.
+
+### Administration
 
-## Script Categories
+*   `permissions`: Audit user and role permissions.
+*   `ownership`: Display table and object ownership.
 
-- **Monitoring** - Database health, locks, replication, bloating
-- **Maintenance** - VACUUM, ANALYZE, statistics collection
-- **Automation** - Health checks, scheduling, alerting
-- **Administration** - Extensions, ownership, constraints, partitions
-- **Optimization** - Index recommendations, HOT updates, missing indexes
-- **Performance** - Query profiling, wait events, resource monitoring
-- **Security** - Permission audits, compliance checks
-- **Troubleshooting** - Diagnostic queries and cheat sheets
-- **Backup & Recovery** - Backup validation and integrity checks
-- **Configuration** - Parameter tuning and analysis
-- **Integration** - Grafana dashboards, Prometheus exporters
+## Incident Response Workflow Example
+
+A customer reports "the database is slow." Here's a typical triage flow using `pgtools`:
+
+1.  **Check for blocking locks.** This is the most common cause of a sudden slowdown.
+    ```bash
+    ./pgtools.sh locks "<conn_string>"
+    ```
+
+2.  **Check current activity.** See what queries are actively running or waiting.
+    ```bash
+    ./pgtools.sh activity "<conn_string>"
+    ```
+
+3.  **Check top queries.** If `pg_stat_statements` is enabled, find out which queries are consuming the most database time historically.
+    ```bash
+    ./pgtools.sh top-queries "<conn_string>"
+    ```
+
+4.  **Check cache hit rate.** A low hit rate points to I/O bottlenecks.
+    ```bash
+    ./pgtools.sh cache-hit "<conn_string>"
+    ```
 
 ## Contributing
 
-Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
+Contributions are welcome! Please see CONTRIBUTING.md for detailed guidelines on how to add new diagnostic scripts.
 
 ## License
 
-See [LICENSE](LICENSE) file for details.
-
-## Support
+This project is licensed under the MIT License - see the LICENSE file for details.
 
-For issues, questions, or contributions, please open an issue in the repository.