Skip to content

Add external compression hypertable config: column_codec #10255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kvc0 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add external compression hypertable config: column_codec #10255

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
937fbd0
Add external compression algorithm
kvc0 Jul 14, 2026
0aea9ef
add timescaledb.compress_column_codec option
kvc0 Jul 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .unreleased/external-compression
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Implements: #1524 Add external compression, letting a type supply its own compress/decompress functions
3 changes: 2 additions & 1 deletion sql/pre_install/insert_data.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@ insert into _timescaledb_catalog.compression_algorithm( id, version, name, descr
( 4, 1, 'COMPRESSION_ALGORITHM_DELTADELTA', 'deltadelta'),
( 5, 1, 'COMPRESSION_ALGORITHM_BOOL', 'bool'),
( 6, 1, 'COMPRESSION_ALGORITHM_NULL', 'null'),
( 7, 1, 'COMPRESSION_ALGORITHM_UUID', 'uuid');
( 7, 1, 'COMPRESSION_ALGORITHM_UUID', 'uuid'),
( 8, 1, 'COMPRESSION_ALGORITHM_EXTERNAL', 'external');

3 changes: 3 additions & 0 deletions sql/updates/latest-dev.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,3 +147,6 @@ GRANT SELECT ON _timescaledb_catalog.chunk TO PUBLIC;
-- END add chunk.relid
--


INSERT INTO _timescaledb_catalog.compression_algorithm( id, version, name, description) values
( 8, 1, 'COMPRESSION_ALGORITHM_EXTERNAL', 'external');
2 changes: 2 additions & 0 deletions sql/updates/reverse-dev.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,3 +250,5 @@ DROP FUNCTION IF EXISTS _timescaledb_functions.policy_compaction_check(JSONB);

DROP FUNCTION IF EXISTS @extschema@.alter_job(job_id INTEGER, schedule_interval INTERVAL, max_runtime INTERVAL, max_retries INTEGER, retry_period INTERVAL, scheduled BOOL, config JSONB, next_start TIMESTAMPTZ, if_exists BOOL, check_config REGPROC, fixed_schedule BOOL, initial_start TIMESTAMPTZ, timezone TEXT, job_name TEXT, config_merge JSONB);


DELETE FROM _timescaledb_catalog.compression_algorithm WHERE id = 8 AND version = 1 AND name = 'COMPRESSION_ALGORITHM_EXTERNAL';
12 changes: 12 additions & 0 deletions tsl/src/compression/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,18 @@ The algorithm checks the cardinality of the values in the compressed batch and b
the cardinality it decides wether it is worth to recompress the batch using the dictionary
compression algorithm. In that case it recompresses and stores the UUIDs as a dictionary.

### External

The external method delegates compression to the column's data type. A type
opts in by providing two SQL functions in its own schema,
`<typename>_compress(<type>[]) RETURNS bytea` and

@akuzm akuzm Jul 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't thought about this feature and don't know if it's feasible/desired. It is a very high-impact area, I'm not sure we have resources now to address it with the care it requires. In general, at this stage it might be best to write a design document instead of code.

One thing catches my attention here: instead of name interpolation, this should probably use the standard Postgres approach with operator classes (access method support functions).

@kvc0 kvc0 Jul 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw a willingness to entertain a contribution in the earlier linked issue, so I figured I’d give it a shot. Thanks for the feedback. You make a great point about operator classes. I’m going to take a look at switching to that kind of approach instead. I’m not that deep on postgres :-)

@kvc0 kvc0 Jul 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, this is much easier than I expected. And it's going to help with upgrade/downgrade. Thanks again for this awesome suggestion

`<typename>_decompress(bytea) RETURNS <type>[]`. It is then selected by
default over dictionary/array for that type. NULLs are stripped into a bitmap
before the compress call and re-inserted after decompression, so these functions
only see non-null values. The returned bytea is stored with TOAST compression
disabled. This lets extension types with internal structure (records, histograms)
compress across rows in ways the generic algorithms cannot.

# Merging chunks while compressing #

## Setup ##
Expand Down
1 change: 1 addition & 0 deletions tsl/src/compression/algorithms/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ set(SOURCES
${CMAKE_CURRENT_SOURCE_DIR}/datum_serialize.c
${CMAKE_CURRENT_SOURCE_DIR}/deltadelta.c
${CMAKE_CURRENT_SOURCE_DIR}/dictionary.c
${CMAKE_CURRENT_SOURCE_DIR}/external.c
${CMAKE_CURRENT_SOURCE_DIR}/gorilla.c
${CMAKE_CURRENT_SOURCE_DIR}/bool_compress.c
${CMAKE_CURRENT_SOURCE_DIR}/null.c
Expand Down
Loading
Loading