Add external compression hypertable config: column_codec - #10255
Conversation
This change lets users define custom functions to do compression
for arbitrary types.
Some custom types are composites, and some come from plugins.
Having the ability to define a codec gives you better control over
compression.
I did this because I needed to define a pgrx type, and the default
array/toast compression did not do as good as I hoped. I spent a
little time on it, and with this change to timescaledb I was able
to get my data type into 33-45% less space. That's a major
difference for me and my small ssds!
The idea for External codecs is basically to extend custom data
types with intrinsic serialization codecs. They're resolved by a
name rather than a whole registry. You make a custom data type,
and 2 functions:
* <schema>.<typename>_compress(<type>[]) RETURNS bytea
* <schema>.<typename>_decompress(bytea) RETURNS <type>[]
When timescale goes to compress a column that is a "fallback"
type, you first try to look up (cached) the codec function pair
for the data type. If it exists, you use it. If it does not
exist, you just fall through and serialize exactly how you do
today.
I serialized a table of custom rust histograms I have via these
functions. That's where I learned the compression is 1/3 to 1/2
the size of toast compression.
By way of example, these are the stubs for a type called
`exphist` in pgrx flavor, to take advantage of the feature in
this PR:
```rust
fn exphist_compress(batch: pgrx::Array<ExpHist>) -> Vec<u8> {
}
fn exphist_decompress(blob: &[u8]) -> Vec<ExpHist> {
}
```
Resolves: timescale#1524
|
@natalya-aksman, @melihmutlu: please review this pull request.
|
|
|
||
| The external method delegates compression to the column's data type. A type | ||
| opts in by providing two SQL functions in its own schema, | ||
| `<typename>_compress(<type>[]) RETURNS bytea` and |
There was a problem hiding this comment.
I haven't thought about this feature and don't know if it's feasible/desired. It is a very high-impact area, I'm not sure we have resources now to address it with the care it requires. In general, at this stage it might be best to write a design document instead of code.
One thing catches my attention here: instead of name interpolation, this should probably use the standard Postgres approach with operator classes (access method support functions).
There was a problem hiding this comment.
I saw a willingness to entertain a contribution in the earlier linked issue, so I figured I’d give it a shot. Thanks for the feedback. You make a great point about operator classes. I’m going to take a look at switching to that kind of approach instead. I’m not that deep on postgres :-)
There was a problem hiding this comment.
oh, this is much easier than I expected. And it's going to help with upgrade/downgrade. Thanks again for this awesome suggestion
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
My concern with this approach is about the compatibility between external algorithm versions. What happens when someone assigns a different/new algorithm for a type? Same question about the migration between the current Timescaledb Array/Dictionary and the external compression? The Timescaledb strategy for this was that we haven't retired any previous compression algorithm from the code, so we can always read all previous algorithms on upgrades. On downgrades, we recompress the data with the algorithm available in the previous version and then we can safely downgrade. How would this work with an external algorithm? |
It’s going to be up to the user to decide compression compatibility. My codec uses a version byte so I can make updates in the future. Broadly speaking, I think all compression codec authors have to consider this: Timescale doesn’t control external users’ deployment schedules, so the user’s versioning contract can only be with the codec author. I think that’s a necessary fallout of external compression. Well, either that or some machinery to force re-compression on update; but I actively do not want that! Decompress old, compress new is a good approach for lazy decompression with a new format generally.
I’m thinking of this more like the chunk time interval setting. That only applies to new chunks (obviously you know all this). So the external compression algorithm should apply to new chunks. Interestingly, you deal with this problem today between the array and dictionary codecs. You write an algorithm byte in the batch header. It’s the same strategy, from timescale’s perspective, as my external algorithm. Depending on the system state, an algorithm is chosen at write time. At read time, the deserializer is chosen by algorithm byte. Array/dictionary/external should all work fine in the same hypertatable by virtue of your batch header.
I think the opclass approach akuzm mentioned has a path forward on this. I should be able to do the same thing for a timescale downgrade. (Read external, recompress array/dictionary) I should have an update this week. Separately, upgrade/downgrade/change of compression algorithm is between a user and their codec author. Thank you both for your thoughtful comments! I realize this change comes a bit out of left field, but I can’t justify paying double to store my histograms as toast without trying to upstream a feature :-) |
| /* Prefer the type's own registered compress/decompress function pair */ | ||
| if (external_codec_lookup(typeoid, NULL, NULL)) | ||
| { | ||
| return COMPRESSION_ALGORITHM_EXTERNAL; | ||
| } |
There was a problem hiding this comment.
Should this behavior be default when the codec exists or be more explicit with a knob like:
alter table metrics
set (
timescaledb.compress_column_codec = 'somecol:some_schema.some_codec'
)There was a problem hiding this comment.
Yeah, that was top of mind for me too, but I was worried the hypertable config section would be harder to merge 😅 than a niche leaf. I will look into this also!
Instead of using types and default opclasses, the EXTERNAL compresison algorithm now uses regular opclasses and only applies when explicitly configured. Users alter or otherwise set the timescaledb.column_codec to a `column_name:some.compression_opclass`. The opclass, as in the previous commit, is a virtual namespace with 2 support functions. (1 and 2, compress and decompress respectively).
kvc0
changed the title
|
@surister -> I have added a @akuzm -> I have switched to using opclass support functions, with amvalidate. It's a much more robust system. I considered skipping it with the tsdb.column_codec change, but it still seemed useful. It's a nice way to declare 1 identifier that has rules, and that can map to multiple functions. It's kind of an "interface" of sorts. This was a great suggestion, thank you! @dbeck It works the same irrespective of EXTERNAL algorithms or dictionary/array. For downgrade, you will have to commit to never retiring the
|
4 participants
This change lets users define custom functions to do compression for arbitrary types.
Some custom types are composites, and some come from plugins. Having the ability to define a codec gives you better control over compression.
I did this because I needed to define a pgrx type, and the default array/toast compression did not do as good as I hoped. I spent a little time on it, and with this change to timescaledb I was able to get my data type into 33-45% less space. That's a major difference for me and my small ssds!
The idea for External codecs is basically to extend custom data types with serialization codecs. They're resolved by configuration after being set up. You make an opclass on the type with 2 functions:
When timescale goes to compress a column, it checks if
column_codecis configured. If so, it uses it to compress the row batch. If column_codec is not configured, it just falls through and serializes via the pre-existing rules.I serialized a table of custom rust histograms I have via these functions. That's where I learned the opportunity for compression is 1/3 to 1/2 the size of toast compression.
By way of example, these are the stubs for a type called
exphistin pgrx flavor, to take advantage of the feature in this PR:Then some sql to set it up:
and in my create table I include a configuration mention:
Resolves: #1524