Add firstlast sparse index type for compression

[antekresic]

Add a new sparse index type that stores the actual first and last values of each compressed batch in sort order, including NULLs. Unlike minmax, which skips NULLs and tracks statistical extremes, firstlast tracks positional boundary values. This enables tracking batch ordering and NULL presence at batch boundaries.

[github-actions]

@melihmutlu, @natalya-aksman: please review this pull request.

Powered by pull-review

[dbeck]

How about dropping and renaming columns? What happens to the corresponding sparse indexes?

[codecov]

Codecov Report

❌ Patch coverage is 88.52459% with 14 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...src/compression/batch_metadata_builder_firstlast.c	87.32%	5 Missing and 4 partials ⚠️
tsl/src/compression/compression.c	81.25%	0 Missing and 3 partials ⚠️
tsl/src/compression/create.c	89.47%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[akuzm]

I like "first" and "last" as the names for the individual boundaries, but maybe we should rename the index to "range" or something?

[antekresic]

I think first last are descriptive enough. Range is particularly deceptive since it only makes sense for the first orderby column, everything else is not really range like.

[svenklemm]

current first/last can only order by a single column, does the firstindex sparse index share that limitation?

[akuzm]

Are they per column? I remember we went over various aspects that influence this decision, i.e. how to plug them into a btree index then. Could you please write a short note of what you ultimately decided and why?

[antekresic]

This is just the first step of introducing these sparse indexes. After this will go into adding them to the btree index which I'm not quite sure how to approach given we use minmax for filtering and we can't really use this for the same purpose.

Thus, this PR is solely focused for the initial step off adding the sparse indexes themselves.

[akuzm]

I'm mostly thinking about the btree we use for ordering. There are other options for storing this, e.g. storing column values, or storing values of record, or even a range of records. But we decided that we want to store per-column values, because we need a btree index on this that matches our "orderby" configuration. And that configuration can use desc/asc nulls first/last, which you can't configure for record fields. And also there's a thing that we can't get away with only storing the "first" or the "last" in the btree, because there are batches where "first" and "last" are the same, and we can't distinguish them by a single boundary. And so on.

This is all non-obvious and we have to get this straight before we release this, so it might make sense to document for the other people looking at the PR.

[antekresic]

@dbeck looks like renaming/dropping isn't fully covered on the existing implementation. Should probably handle that correctly in a separate PR.

[dbeck]

@dbeck looks like renaming/dropping isn't fully covered on the existing implementation. Should probably handle that correctly in a separate PR.

I think the full lifecycle should be in the same PR, otherwise we may never get the rest implemented.

[antekresic]

I don't think its smart for every sparse index we add to need to manually handle dropping/renaming of columns, it should be part of the generic codebase that handles it automatically based on the compression settings.

[akuzm]

I don't think its smart for every sparse index we add to need to manually handle dropping/renaming of columns, it should be part of the generic codebase that handles it automatically based on the compression settings.

I thought this worked... At the moment changing the indexes just requires full recompression, and this just recreates the sparse indexes, right? And @Poroma-Banerjee is working on a smarter approach that would allow us to change this more granularly.

[dbeck]

I don't think its smart for every sparse index we add to need to manually handle dropping/renaming of columns, it should be part of the generic codebase that handles it automatically based on the compression settings.

Both for dropping and renaming columns we need to update the compression settings and we need to handle the metadata column changes as well (rename the metadata column or dropping it).

In the absence of generic handling we can't just YOLO new metdata columns and new settings field. I do not thin this is right.

[dbeck]

I don't think its smart for every sparse index we add to need to manually handle dropping/renaming of columns, it should be part of the generic codebase that handles it automatically based on the compression settings.

I thought this worked... At the moment changing the indexes just requires full recompression, and this just recreates the sparse indexes, right? And @Poroma-Banerjee is working on a smarter approach that would allow us to change this more granularly.

@akuzm : do you mean the case when the change is triggered by an ALTER table SET command? I think we also need to handle the the ALTER .. DROP and ALTER .. RENAME cases. These don't need a recompression.

[Poroma-Banerjee]

https://github.com/timescale/eng-database/issues/772 -> first(value, time) and last(value, time) using ColumnarIndexScan

Based on above naming might probably clash with other sparse indexes that are to be introduced. Actually I'm confused with whether it is the same thing.

• minmax of column x. will store the min(x) and max(x) regardless of batch sort order. excludes NULLS
• firstlast of column x. will store be value of x at min(time) and value of x at max(time). This will be batch boundary and can be NULL.
• what if orderby is NULLABLE?
  Then, firstlast of column 'time'. will store first as min(time) and last as max(time) or NULL if last tuple of the Batch was NULL.

In that case firstlast could be leveraged to support first(value, time) and last(value, time) using ColumnarIndexScan because it sounds like the same thing

[svenklemm]

is there value in having first/last be separate? user might only need one of the two

[antekresic]

Devils advocate: you might say the same for minmax. I don't think the increased complexity of separating them out warrants the storage savings.

[natalya-aksman]

I wonder if we could consider optionally saving has_nulls info which we obtain while creating BatchMetadataBuilderMinMax builder here.

We can save it in a index like _ts_meta_hasnulls_1 to go with _ts_meta_min_1 and _ts_meta_max_1 if the (1st in this case) orderby column is not guaranteed to be NOT NULL.
This is all the info we need to correctly compare tuples when recompressing, as match_tuple_batch has access to nulls_is_first info for each orderby column and can correctly merge if it knows that orderby column has NULLs and knows whether they come first or last.

We can add this metadata column automatically along with minmax metadata columns if orderby column is not guaranteed to be NOT NULL. It'll be just an extra boolean column in compressed chunk schema.