Skip to content

Commit 3167542

Browse files
authored
feat: expose PGLZ compression bindings + add pglz_inspect example (#2326)
## Summary Adds a safe Rust wrapper around PostgreSQL's in-tree PGLZ compressor (common/pg_lzcompress.h) and ships a companion example extension that uses it to advise DBAs on column-level compression decisions. ## Motivation pglz_compress / pglz_decompress / pglz_maximum_compressed_size are stable, palloc-free, elog-free C functions that ship with every supported PostgreSQL version. Extension authors today have no first-class way to call them from pgrx — they have to either reach into pg_sys and write their own unsafe, or shell out to SQL and pay the round-trip. Exposing a vetted safe wrapper lets the ecosystem build TOAST-aware tooling, custom storage layouts, compression-aware indexes. ## Benefit Real-world utility out of the box — pglz_inspect answers "should I compress this column?" with one query, lowering the barrier for DBAs evaluating PG14+'s per-column compression.
1 parent b28919b commit 3167542

12 files changed

Lines changed: 1024 additions & 0 deletions

File tree

Cargo.lock

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#LICENSE Portions Copyright 2019-2021 ZomboDB, LLC.
2+
#LICENSE
3+
#LICENSE Portions Copyright 2021-2023 Technology Concepts & Design, Inc.
4+
#LICENSE
5+
#LICENSE Portions Copyright 2023-2023 PgCentral Foundation, Inc. <contact@pgcentral.org>
6+
#LICENSE
7+
#LICENSE All rights reserved.
8+
#LICENSE
9+
#LICENSE Use of this source code is governed by the MIT license that can be found in the LICENSE file.
10+
11+
[package]
12+
name = "pglz_inspect"
13+
version = "0.0.0"
14+
edition.workspace = true
15+
rust-version.workspace = true
16+
publish = false
17+
18+
[lib]
19+
crate-type = ["cdylib"]
20+
21+
[features]
22+
default = ["pg13"]
23+
pg13 = ["pgrx/pg13", "pgrx-tests/pg13"]
24+
pg14 = ["pgrx/pg14", "pgrx-tests/pg14"]
25+
pg15 = ["pgrx/pg15", "pgrx-tests/pg15"]
26+
pg16 = ["pgrx/pg16", "pgrx-tests/pg16"]
27+
pg17 = ["pgrx/pg17", "pgrx-tests/pg17"]
28+
pg18 = ["pgrx/pg18", "pgrx-tests/pg18"]
29+
pg_test = []
30+
31+
[dependencies]
32+
pgrx = { path = "../../pgrx" }
33+
34+
[dev-dependencies]
35+
pgrx-tests = { path = "../../pgrx-tests" }
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# pglz_inspect
2+
3+
A pgrx-based PostgreSQL extension that helps DBAs decide whether to enable PGLZ compression on a column. Samples real data, runs it through the in-tree `pglz_compress` implementation, and reports compression ratio, acceptance rate, estimated disk savings, and an actionable recommendation.
4+
Supports PostgreSQL 13–18.
5+
6+
## Install
7+
8+
```bash
9+
cargo pgrx install --package pglz_inspect
10+
psql -c 'CREATE EXTENSION pglz_inspect;'
11+
```
12+
13+
## Functions
14+
15+
| Function | Purpose |
16+
|---|---|
17+
| `pglz_size(bytea)` | Probe a single value: raw size, compressed size, ratio, accepted. |
18+
| `pglz_analyze_column(regclass, text, sample_size, strategy)` | Sample N rows; return aggregate stats and estimated savings (uses `pg_class.reltuples`). |
19+
| `pglz_ratio_histogram(regclass, text, sample_size)` | Distribution of per-row compression ratios across 5 buckets + `incompressible`. |
20+
| `pglz_recommend(regclass, text, sample_size)` | One-line RECOMMEND / MARGINAL / SKIP verdict with ready-to-run DDL. |
21+
22+
## Demo
23+
24+
```sql
25+
CREATE EXTENSION pglz_inspect;
26+
27+
CREATE TABLE events AS
28+
SELECT i AS id,
29+
('{"user":'||i||',"action":"click","meta":'||repeat('"x",',50)||'1}') AS payload
30+
FROM generate_series(1, 10000) i;
31+
32+
33+
SELECT * FROM pglz_ratio_histogram('events'::regclass, 'payload');
34+
-- Example output (numbers vary by data):
35+
-- bucket | row_count
36+
-- -----------------+-----------
37+
-- 0.0-0.2 | 9876
38+
-- 0.2-0.4 | 124
39+
-- incompressible | 0
40+
41+
SELECT pglz_recommend('events'::regclass, 'payload');
42+
-- Example output (numbers vary by data):
43+
-- RECOMMEND: PGLZ saves ~83% on events.payload (1000 of 1000 sampled rows accepted).
44+
-- Run: ALTER TABLE events ALTER COLUMN payload SET COMPRESSION pglz;
45+
```
46+
47+
## Histogram bucket semantics
48+
49+
`pglz_ratio_histogram` reports `compressed_size / raw_size` per row, bucketed:
50+
51+
```
52+
┌───────────┬────────────────┐
53+
│ ratio │ bucket │
54+
├───────────┼────────────────┤
55+
│ 0.00-0.20 │ 0.0-0.2 │ ← excellent: ~80%+ saved
56+
├───────────┼────────────────┤
57+
│ 0.20-0.40 │ 0.2-0.4 │ ← good
58+
├───────────┼────────────────┤
59+
│ 0.40-0.60 │ 0.4-0.6 │ ← moderate
60+
├───────────┼────────────────┤
61+
│ 0.60-0.80 │ 0.6-0.8 │ ← weak
62+
├───────────┼────────────────┤
63+
│ 0.80-1.00 │ 0.8-1.0 │ ← negligible
64+
├───────────┼────────────────┤
65+
│ PGLZ rejust │ incompressible │ ← rejected by PGLZ heuristics
66+
└───────────┴────────────────┘
67+
```
68+
69+
Highly repetitive data clusters in `0.0-0.2`; short / high-entropy data lands in `incompressible`. A two-peak histogram (`0.0-0.2` + `incompressible`) is common and means the column mixes very-compressible and very-random rows.
70+
71+
## Mixed-distribution example
72+
73+
The default demo data is too uniform to populate the middle buckets. To see a spread across all buckets, use semi-repetitive data:
74+
75+
```sql
76+
DROP TABLE IF EXISTS mixed;
77+
78+
CREATE TABLE mixed AS
79+
SELECT i AS id,
80+
CASE (i % 5)
81+
-- 0.0-0.2: heavy repetition
82+
WHEN 0 THEN repeat('aaaaa', 100)
83+
-- 0.2-0.4: structured JSON, repeated field names + varied values
84+
WHEN 1 THEN '{"user_id":'||i||',"event_type":"page_view","timestamp":"2024-01-'||(i%28+1)||'","session":"'||md5(i::text)||'"}'
85+
-- 0.4-0.6: short repeated fragment + variable suffix
86+
WHEN 2 THEN repeat(md5(i::text), 3) || md5((i+1)::text)
87+
-- 0.6-0.8: mostly random, small repeating prefix
88+
WHEN 3 THEN 'log_entry_' || md5(i::text) || md5((i+1)::text) || md5((i+2)::text)
89+
-- incompressible: pure randomness
90+
ELSE (SELECT string_agg(md5(random()::text), '') FROM generate_series(1, 4))
91+
END AS payload
92+
FROM generate_series(1, 5000) i;
93+
94+
SELECT * FROM pglz_ratio_histogram('mixed'::regclass, 'payload', 5000);
95+
```
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
comment = 'pglz_inspect: PGLZ compression analysis for DBAs'
2+
default_version = '@CARGO_VERSION@'
3+
module_pathname = 'pglz_inspect'
4+
relocatable = false
5+
superuser = false

0 commit comments

Comments
 (0)