Commit 1237fd6
committed
feature: asynchronously update file statistics during vacuum
Previously, statistics (min-max, sum, count, etc.) were computed synchronously
during data insertion, causing significant slowdowns due to heavy computational
overhead.
This change introduces an asynchronous approach to maintain statistics:
- Add a GUC parameter to control statistics collection during writes (disabled by default)
- Skip statistics computation during INSERT to ensure fast writes
- Update statistics asynchronously during VACUUM on PAX tables by scanning file metadata
- Re-read files and refresh statistics only when metadata indicates they are stale
- Vacuum which data files have been marked for deletion
```
create table t1(c1 int, c2 int, c3 int, c4 int, c5 int, c6 int) using pax with(minmax_columns='c1,c2,c3,c4,c5,c6');
set pax.enable_sync_collect_stats to on; -- collect stats synchronously
insert into t1 select i,i,i,i,i,i from generate_series(1,1000000) i;
INSERT 0 1000000
Time: 2733.731 ms (00:02.734)
create table t2(c1 int, c2 int, c3 int, c4 int, c5 int, c6 int) using pax;
insert into t2 select i,i,i,i,i,i from generate_series(1,1000000) i;
INSERT 0 1000000
Time: 1816.836 ms (00:01.817)
```1 parent da0b9d4 commit 1237fd6
File tree
81 files changed
+1657
-676
lines changed- contrib/pax_storage
- expected
- statistics
- sql
- statistics
- src
- cpp
- access
- catalog
- clustering
- cmake
- comm
- storage
- orc
- test/isolation2/expected/pax
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
81 files changed
+1657
-676
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
| 2 | + | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
| |||
295 | 296 | | |
296 | 297 | | |
297 | 298 | | |
| 299 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
223 | 224 | | |
224 | 225 | | |
225 | 226 | | |
| 227 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
| |||
223 | 224 | | |
224 | 225 | | |
225 | 226 | | |
| 227 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
735 | 736 | | |
736 | 737 | | |
737 | 738 | | |
| 739 | + | |
738 | 740 | | |
739 | 741 | | |
740 | 742 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
742 | 743 | | |
743 | 744 | | |
744 | 745 | | |
| 746 | + | |
745 | 747 | | |
746 | 748 | | |
747 | 749 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
1063 | 1064 | | |
1064 | 1065 | | |
1065 | 1066 | | |
| 1067 | + | |
1066 | 1068 | | |
1067 | 1069 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
938 | 939 | | |
939 | 940 | | |
940 | 941 | | |
| 942 | + | |
941 | 943 | | |
942 | 944 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| |||
738 | 739 | | |
739 | 740 | | |
740 | 741 | | |
| 742 | + | |
741 | 743 | | |
742 | 744 | | |
743 | 745 | | |
| |||
0 commit comments