Skip to content

Commit 7541e93

Browse files
committed
- v0.2.0
- add yum package and linux service definition - add a 'skip' flag into query config - fix `pgbouncer_up` metrics - add conf reload support
1 parent 00d88ae commit 7541e93

14 files changed

+512
-103
lines changed

.gitignore

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,26 @@
1+
# binary files
12
bin/
3+
pg_exporter
4+
5+
# IDE files
26
.vscode/
37
.idea/
48
pg_exporter.iml
9+
10+
# rpm files
11+
rpmbuild/
12+
make-rpm.sh
13+
14+
# go files
515
go.mod
616
go.sum
717
vendor/
8-
pg_exporter
18+
19+
# mac os files
920
.DS_Store
21+
22+
# test files
1023
pg_exporter_test.go
1124
test/
1225
deploy/
1326
upload.sh
14-
15-
conf/adhoc/

Makefile

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,25 +4,45 @@ build:
44
go build -o pg_exporter
55

66
clean:
7-
rm -rf bin/pg_exporter pg_exporter conf.tar.gz pg_exporter.yaml
7+
rm -rf pg_exporter conf.tar.gz
88

9-
release-darwin: clean
9+
release-darwin:
1010
CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -a -ldflags '-extldflags "-static"' -o pg_exporter
1111
upx pg_exporter
12-
tar -cf bin/pg_exporter_v$(VERSION)_darwin-amd64.tar.gz pg_exporter
13-
rm -rf pg_exporter
12+
mkdir -p bin/pg_exporter_v$(VERSION)_darwin-amd64
13+
mv -f pg_exporter bin/pg_exporter_v$(VERSION)_darwin-amd64/
14+
cp -f pg_exporter.yaml bin/pg_exporter_v$(VERSION)_darwin-amd64/
15+
tar -cf bin/pg_exporter_v$(VERSION)_darwin-amd64.tar.gz -C bin pg_exporter_v$(VERSION)_darwin-amd64
16+
rm -rf bin/pg_exporter_v$(VERSION)_darwin-amd64
1417

1518
release-linux: clean
1619
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags '-extldflags "-static"' -o pg_exporter
1720
upx pg_exporter
18-
tar -cf bin/pg_exporter_v$(VERSION)_linux-amd64.tar.gz pg_exporter
19-
rm -rf pg_exporter
21+
mkdir -p bin/pg_exporter_v$(VERSION)_linux-amd64
22+
mv -f pg_exporter bin/pg_exporter_v$(VERSION)_linux-amd64/
23+
cp -f pg_exporter.yaml bin/pg_exporter_v$(VERSION)_linux-amd64/
24+
cp -f service/pg_exporter.default bin/pg_exporter_v$(VERSION)_linux-amd64/
25+
cp -f service/pg_exporter.service bin/pg_exporter_v$(VERSION)_linux-amd64/
26+
tar -cf bin/pg_exporter_v$(VERSION)_linux-amd64.tar.gz -C bin pg_exporter_v$(VERSION)_linux-amd64
27+
rm -rf bin/pg_exporter_v$(VERSION)_linux-amd64
2028

2129
release-windows: clean
2230
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -a -ldflags '-extldflags "-static"' -o pg_exporter
2331
upx pg_exporter
24-
tar -cf bin/pg_exporter_v$(VERSION)_windows-amd64.tar.gz pg_exporter
25-
rm -rf pg_exporter
32+
mkdir -p bin/pg_exporter_v$(VERSION)_windows-amd64
33+
mv -f pg_exporter bin/pg_exporter_v$(VERSION)_windows-amd64/
34+
cp -f pg_exporter.yaml bin/pg_exporter_v$(VERSION)_windows-amd64/
35+
tar -cf bin/pg_exporter_v$(VERSION)_windows-amd64.tar.gz -C bin pg_exporter_v$(VERSION)_windows-amd64
36+
rm -rf bin/pg_exporter_v$(VERSION)_windows-amd64
37+
38+
rpm:
39+
./make-rpm.sh
40+
41+
install: build
42+
sudo install -m 0755 pg_exporter /usr/bin/pg_exporter
43+
44+
uninstall:
45+
sudo rm -rf /usr/bin/pg_exporter
2646

2747
conf:
2848
rm -rf pg_exporter.yaml
@@ -41,7 +61,7 @@ curl:
4161
upload:
4262
./upload.sh
4363

44-
release: release-linux release-darwin release-windows
64+
release: clean conf release-linux release-darwin release-windows
4565

4666

47-
.PHONY: build clean release-linux release-darwin release-windows release linux docker run curl conf upload
67+
.PHONY: build clean release-linux release-darwin release-windows rpm release linux docker run curl conf upload

README.md

Lines changed: 100 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,15 @@
22

33
[Prometheus](https://prometheus.io/) [exporter](https://prometheus.io/docs/instrumenting/exporters/) for [PostgreSQL](https://www.postgresql.org) metrics. **Gives you complete insight on your favourate elephant!**
44

5-
Latest binaries can be found on [release](https://github.com/Vonng/pg_exporter/releases) page. [Support](pg_exporter.yaml) for PostgreSQL 10+ & Pgbouncer 1.9+ is provided ([`conf`](conf/)). Lower version support could be add in the future, or you can provide your own conf to achieve it.
6-
7-
Latest version is v0.1.2 Here are how self monitoring dashboards looks like:
8-
9-
![](doc/pg-exporter.png)
5+
Latest binaries & rpm can be found on [release](https://github.com/Vonng/pg_exporter/releases) page. Supported pg version: PostgreSQL 9.4+ & Pgbouncer 1.8+
106

7+
The latest version of `pg_exporter` is v0.2.0
118

9+
## Feature
1210

13-
## Features
14-
15-
* Support Postgres & Pgbouncer (pgbouncer mode is enabled when target dbname is `pgbouncer`)
16-
* Fine-grained execution control (Tags Filter, Facts Filter, Version Filter, Timeout, Cache, etc...)
11+
* Support both Postgres & Pgbouncer
1712
* Flexible: Almost all metrics are defined in customizable config files with SQL query.
13+
* Fine-grained execution control (Tags Filter, Facts Filter, Version Filter, Timeout, Cache, etc...)
1814
* Dynamic Planning: User could provide multiple branches of a metric queries. Queries matches server version & fact & tag will be actually installed.
1915
* Configurable caching policy & query timeout
2016
* Rich metrics about `pg_exporter` itself.
@@ -24,12 +20,14 @@ Latest version is v0.1.2 Here are how self monitoring dashboards looks like:
2420

2521

2622

23+
(pgbouncer mode is enabled when target dbname is `pgbouncer`)
24+
2725
## Quick Start
2826

2927
To run this exporter, you will need two things
3028

3129
* **Where** to scrape: A postgres or pgbouncer URL given via `PG_EXPORTER_URL` or `--url`
32-
* **What** to scrape: A path to config file or directory, by default `./pg_exporter.yaml`
30+
* **What** to scrape: A path to config file or directory, by default `./pg_exporter.yaml` or `/etc/pg_exporter`
3331

3432
```bash
3533
export PG_EXPORTER_URL='postgres://postgres:password@localhost:5432/postgres'
@@ -116,6 +114,16 @@ Or [download](https://github.com/Vonng/pg_exporter/releases) latest prebuilt bin
116114

117115

118116

117+
## Deploy
118+
119+
A redhat7 rpm is provided on release page. Which include
120+
121+
* [`/etc/default/pg_exporter`](service/pg_exporter.default)
122+
* [`/etc/pg_exporter/pg_exporter.yaml`](service/pg_exporter.default)
123+
* [`/usr/pg_exporter`](service/pg_exporter.default)
124+
125+
126+
119127
## Config
120128

121129
Configs are core part of `pg_exporter`. Actually this project contains more lines of YAML than go.
@@ -126,9 +134,7 @@ Configs are core part of `pg_exporter`. Actually this project contains more line
126134

127135
### built-in configs
128136

129-
Dashboard for monitoring pg_exporter itself: [`pg-exporter.json`](monitor/pg-exporter.json)
130137

131-
![](doc/pg-exporter.png)
132138

133139
Current `pg_exporter` is ship with 32 built-in metrics queries.
134140

@@ -162,90 +168,98 @@ Current `pg_exporter` is ship with 32 built-in metrics queries.
162168
* [pg_index_bloat](conf/127-pg_index_bloat.yaml)
163169
* [pg_func](conf/128-pg_func.yaml)
164170
* [pgbouncer_list](conf/129-pgbouncer_list.yaml)
165-
* [pgbouncer_stat](conf/130-pgbouncer_stat.yaml)
166-
* [pgbouncer_database](conf/131-pgbouncer_database.yaml)
167-
* [pgbouncer_pool](conf/132-pgbouncer_pool.yaml)
171+
* [pgbouncer_database](conf/130-pgbouncer_database.yaml)
172+
* [pgbouncer_pool](conf/131-pgbouncer_pool.yaml)
173+
* [pgbouncer_stat](conf/132-pgbouncer_stat.yaml)
168174

169175
`pg_exporter` will generate approximately 200~300 metrics for completely new database cluster. For a real-world database with 10 ~ 100 tables, it may generate serveral 1k ~ 10k metrics. You may need modifying or disable some database-level metrics on database with serveral thousands or more tables in order to complete scrape in time.
170176

171177
Config files are using YAML format, there are lots of examples in the [conf](https://github.com/Vonng/pg_exporter/tree/master/conf) dir. and here is a [sample](conf/100-doc.txt) config.
172178

173179
```yaml
174-
# pg_primary_only: <---- Branch name, distinguish different branch of a metric query
175-
# name: pg <---- actual Query name, used as metric prefix, will set to branch if not provide
176-
# desc: PostgreSQL basic information (on primary) <---- query description
177-
# query: | <---- query string
178-
#
179-
# SELECT extract(EPOCH FROM CURRENT_TIMESTAMP) AS timestamp,
180-
# pg_current_wal_lsn() - '0/0' AS lsn,
181-
# pg_current_wal_insert_lsn() - '0/0' AS insert_lsn,
182-
# pg_current_wal_lsn() - '0/0' AS write_lsn,
183-
# pg_current_wal_flush_lsn() - '0/0' AS flush_lsn,
184-
# extract(EPOCH FROM now() - pg_postmaster_start_time()) AS uptime,
185-
# extract(EPOCH FROM now() - pg_conf_load_time()) AS conf_reload_time,
186-
# pg_is_in_backup() AS is_in_backup,
187-
# extract(EPOCH FROM now() - pg_backup_start_time()) AS backup_time;
188-
#
189-
# <---- following field are [OPTIONAL], control execution policy
190-
# ttl: 10 <---- cache ttl: how long will exporter cache it's result. set to 0 to disable cache
191-
# timeout: 0.1 <---- timeout: in seconds, query execeed this will be canceled. default is 0.1, set to -1 to disable timeout
192-
# min_version: 100000 <---- minimal supported version in server version number format, e.g 120001 = 12.1, 090601 = 9.6.1
193-
# max_version: 130000 <---- maximal supported version in server version number format, boundary not include
194-
# fatal: false <---- if query marked fatal fail, this scrape will abort immidiately
195-
#
196-
# tags: [cluster, primary] <---- tags consist of one or more string, which could be:
197-
# * 'cluster' marks this query as cluster level, so it will only execute once for same PostgreSQL Server
198-
# * 'primary' marks this query can only run on a master instance (will not execute if pg_is_in_recovery())
199-
# * 'standby' marks this query can only run on a recoverying instance (will execute if pg_is_in_recovery())
200-
# * some special tag prefix will have special effect:
201-
# * 'dbname:<dbname>' means this query will only execute on database with name '<dbname>'
202-
# * 'username:<user>' means this query will only execute when connect with user '<user>'
203-
# * 'extension:<extname>' means this query will only execute when extension '<extname>' is installed
204-
# * 'schema:<nspname>' means this query will only execute when schema '<nspname>' exist
205-
# * 'not:<negtag>' means this query will only execute when exporter is launch without tag '<negtag>'
206-
# * '<tag>' means this query will only execute when exporter is launch with tag '<tag>'
207-
# (tag could not be cluster,primary,standby or have special prefix)
208-
#
209-
#
210-
# metrics: <---- this is a list of returned columns, each column must have a name, usage, could have an alias and description
211-
# - timestamp: <---- this is column name, should be exactly same as returned column name
212-
# rename: ts <---- rename is optional, will use this alias instead of column name
213-
# usage: GAUGE <---- usage could be
214-
# * DISCARD: completely ignore this field
215-
# * LABEL: use columnName:columnValue as a label in result
216-
# * GAUGE: use this column as a metric, which is '<query.name>_<column.name>{<labels>} column.value'
217-
# * COUNTER: same as GAUGE, except it is a counter.
218-
#
219-
# description: database current timestamp
220-
# - lsn:
221-
# usage: COUNTER
222-
# description: log sequence number, current write location (on primary)
223-
# - insert_lsn:
224-
# usage: COUNTER
225-
# description: primary only, location of current wal inserting
226-
# - write_lsn:
227-
# usage: COUNTER
228-
# description: primary only, location of current wal writing
229-
# - flush_lsn:
230-
# usage: COUNTER
231-
# description: primary only, location of current wal syncing
232-
# - uptime:
233-
# usage: GAUGE
234-
# description: seconds since postmaster start
235-
# - conf_reload_time:
236-
# usage: GAUGE
237-
# description: seconds since last configuration reload
238-
# - is_in_backup:
239-
# usage: GAUGE
240-
# description: 1 if backup is in progress
241-
# - backup_time:
242-
# usage: GAUGE
243-
# description: seconds since current backup start. null if don't have one
180+
#┃ Query Example
181+
#┣┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
182+
#┃ pg_primary_only: <---- Branch name, distinguish different branch of a metric query
183+
#┃ name: pg <---- Query's real name, used as metric prefix, set to Branch Name by default
184+
#┃ desc: PostgreSQL basic information (on primary) <---- query description
185+
#┃ query: | <---- query string
186+
#
187+
#┃ SELECT extract(EPOCH FROM CURRENT_TIMESTAMP) AS timestamp,
188+
#┃ pg_current_wal_lsn() - '0/0' AS lsn,
189+
#┃ pg_current_wal_insert_lsn() - '0/0' AS insert_lsn,
190+
#┃ pg_current_wal_lsn() - '0/0' AS write_lsn,
191+
#┃ pg_current_wal_flush_lsn() - '0/0' AS flush_lsn,
192+
#┃ extract(EPOCH FROM now() - pg_postmaster_start_time()) AS uptime,
193+
#┃ extract(EPOCH FROM now() - pg_conf_load_time()) AS conf_reload_time,
194+
#┃ pg_is_in_backup() AS is_in_backup,
195+
#┃ extract(EPOCH FROM now() - pg_backup_start_time()) AS backup_time;
196+
#
197+
#┃ <---- following field are [OPTIONAL], control execution policy
198+
#┃ ttl: 10 <---- cache ttl: how long will exporter cache it's result. set to 0 to disable cache
199+
#┃ timeout: 0.1 <---- timeout: in seconds, query execeed this will be canceled. default is 0.1 (100ms), set to -1 to disable timeout
200+
#┃ min_version: 100000 <---- minimal supported version in server version number format, e.g 120001 = 12.1, 090601 = 9.6.1
201+
#┃ max_version: 130000 <---- maximal supported version in server version number format, boundary not include
202+
#┃ fatal: false <---- if query marked fatal fail, this scrape will abort immediately
203+
#┃ skip: false <---- query marked skip will be omitted (during planning)
204+
#
205+
#┃ tags: [cluster, primary] <---- tags consist of one or more string, which could be:
206+
#┃ * 'cluster' marks this query as cluster level, so it will only execute once for same PostgreSQL Server
207+
#┃ * 'primary' marks this query can only run on a master instance (will not execute if pg_is_in_recovery())
208+
#┃ * 'standby' marks this query can only run on a recovering instance (will execute if pg_is_in_recovery())
209+
#┃ * some special tag prefix will have special effect:
210+
#┃ * 'dbname:<dbname>' means this query will only execute on database with name '<dbname>'
211+
#┃ * 'username:<user>' means this query will only execute when connect with user '<user>'
212+
#┃ * 'extension:<extname>' means this query will only execute when extension '<extname>' is installed
213+
#┃ * 'schema:<nspname>' means this query will only execute when schema '<nspname>' exist
214+
#┃ * 'not:<negtag>' means this query will only execute when exporter is launch without tag '<negtag>'
215+
#┃ * '<tag>' means this query will only execute when exporter is launch with tag '<tag>'
216+
#┃ (tag could not be cluster,primary,standby or have special prefix)
217+
#
218+
#
219+
#┃ metrics: <---- this is a list of returned columns, each column must have a name, usage, could have an alias and description
220+
#┃ - timestamp: <---- this is column name, should be exactly same as returned column name
221+
#┃ rename: ts <---- rename is optional, will use this alias instead of column name
222+
#┃ usage: GAUGE <---- usage could be
223+
#┃ * DISCARD: completely ignore this field
224+
#┃ * LABEL: use columnName:columnValue as a label in result
225+
#┃ * GAUGE: use this column as a metric, which is '<query.name>_<column.name>{<labels>} column.value'
226+
#┃ * COUNTER: same as GAUGE, except it is a counter.
227+
#
228+
#┃ description: database current timestamp
229+
#┃ - lsn:
230+
#┃ usage: COUNTER
231+
#┃ description: log sequence number, current write location (on primary)
232+
#┃ - insert_lsn:
233+
#┃ usage: COUNTER
234+
#┃ description: primary only, location of current wal inserting
235+
#┃ - write_lsn:
236+
#┃ usage: COUNTER
237+
#┃ description: primary only, location of current wal writing
238+
#┃ - flush_lsn:
239+
#┃ usage: COUNTER
240+
#┃ description: primary only, location of current wal syncing
241+
#┃ - uptime:
242+
#┃ usage: GAUGE
243+
#┃ description: seconds since postmaster start
244+
#┃ - conf_reload_time:
245+
#┃ usage: GAUGE
246+
#┃ description: seconds since last configuration reload
247+
#┃ - is_in_backup:
248+
#┃ usage: GAUGE
249+
#┃ description: 1 if backup is in progress
250+
#┃ - backup_time:
251+
#┃ usage: GAUGE
252+
#┃ description: seconds since current backup start. null if don't have one
253+
#┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
244254
```
245255

246256

247257

248-
## Visualize
258+
## VisualizATION
259+
260+
![](doc/pg-exporter.png)
261+
262+
249263

250264
Users could visualize these metrics via fancy grafana dashboards.
251265

conf/100-doc.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
#┃ min_version: 100000 <---- minimal supported version in server version number format, e.g 120001 = 12.1, 090601 = 9.6.1
3434
#┃ max_version: 130000 <---- maximal supported version in server version number format, boundary not include
3535
#┃ fatal: false <---- if query marked fatal fail, this scrape will abort immediately
36+
#┃ skip: false <---- query marked skip will be omitted (during planning)
3637
#
3738
#┃ tags: [cluster, primary] <---- tags consist of one or more string, which could be:
3839
#┃ * 'cluster' marks this query as cluster level, so it will only execute once for same PostgreSQL Server

conf/117-pg_query.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ pg_query:
6161
6262
ttl: 100
6363
timeout: 2
64+
skip: false
6465
tags: [cluster, extension:pg_stat_statements , schema:monitor, not:shard, not:bulky]
6566
min_version: 090400
6667

conf/123-pg_table_size.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ pg_table_size:
3535
3636
ttl: 100
3737
timeout: 1
38+
skip: false
3839
tags: [not:bulky]
3940
min_version: 100000
4041

conf/125-pg_table_bloat.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ pg_table_bloat:
5757
5858
ttl: 100
5959
timeout: 1
60+
skip: false
6061
tags: [not:bulky]
6162
min_version: 090400
6263

conf/127-pg_index_bloat.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ pg_index_bloat:
6060
6161
ttl: 100
6262
timeout: 1
63+
skip: false
6364
tags: [not:bulky]
6465
min_version: 090400
6566

0 commit comments

Comments
 (0)