Add back config to toggle the preservation of timestamps in consolidated fragments

[ypatia]

#3267 erroneously removed the option to perform consolidation without timestamps [sc-18605]. This PR restores that config as the option to not retain timestamps when consolidating is still a valuable one in many cases :

• heavily modified datasets (many updates) which want to reclaim space and do not need time-travel
• code that wants to benefit from cell count metadata on the fragments (and needs no-dups semantics)
• the potential future ability to improve read performance of no-dup arrays by optimizing the reader (e.g., using the sparse allows-dups reader)

Note: I have left the default configuration as "with timestamps", but I have also implemented and tested changing the default to "without timestamps" and everything works fine. I can toggle the default to whatever the requirement is very easily. I just don't know what the requirement is :)

Fixes [CORE-134]

---

TYPE: IMPROVEMENT
DESC: Add back config to toggle the preservation of timestamps in consolidated fragments

[bkmartinjr]

The choice of default (ie., maintain current behavior by default) seems reasonable to me.

[teo-tsirpanis]

Thanks!

I believe we should make a plan to disable this feature by default and eventually remove it. As a concept, consolidation is fundamentally incompatible with time travelling and adding a timestamps pseudo-attribute was a failed attempt to reconcile them.

[ypatia]

Thanks!

I believe we should make a plan to disable this feature by default and eventually remove it. As a concept, consolidation is fundamentally incompatible with time travelling and adding a timestamps pseudo-attribute was a failed attempt to reconcile them.

I don't know the background story on why this feature was introduced in the first place, I can imagine though that it was requested to meet some need. If the way it has been implemented is optimal or not is another story.
However, I agree in principle that consolidation means mostly I don't care about time-traveling, that's why I was wondering and asked to clarify what the default value of this config toggle should be. If 95% of the cases we don't want to have timestamps in consolidated fragments then it'd make sense to change the default value to false. cc @ihnorton

[rroelke]

It's been three years since #3267 was merged so we should definitely be suspicious as to whether the behavior which is being re-exposed here is still correct. I'd really like to see some test cases which validate query results, and also validate that consolidation happened in the expected way using the fragment metadata.

[rroelke]

CC @nickvigilante

I think this comment gets pulled into docs, right?

This is not specific enough, especially if it is user-facing documentation. There's no description here of what the consolidation result actually looks like for the different options, which I think is really important given that one of the options results in something which might qualify as "data loss" for an unwitting customer.

[rroelke]

Okay, I'm leaving my comment for posterity, but I see that there is more specific documentation in config_api_external.h.

However, those docs aren't specific enough for my liking, the options should be annotated with a brief description of what happens to duplicate coordinates in consolidated fragments.

[nickvigilante]

I primarily pull what's in these two files:

• https://github.com/TileDB-Inc/TileDB/blob/2b2a43459785f56a1b73c0558028c7ed8f219d77/tiledb/api/c_api/config/config_api_external.h
• https://github.com/TileDB-Inc/TileDB/blob/2b2a43459785f56a1b73c0558028c7ed8f219d77/tiledb/sm/cpp_api/config.h

[rroelke]

It looks like the the only usage of this is "if false, consolidation_with_timestamps_supported returns false". What's the connection to overlapping fragments?

[rroelke]

Thanks!
I believe we should make a plan to disable this feature by default and eventually remove it. As a concept, consolidation is fundamentally incompatible with time travelling and adding a timestamps pseudo-attribute was a failed attempt to reconcile them.

I don't know the background story on why this feature was introduced in the first place, I can imagine though that it was requested to meet some need. If the way it has been implemented is optimal or not is another story. However, I agree in principle that consolidation means mostly I don't care about time-traveling, that's why I was wondering and asked to clarify what the default value of this config toggle should be. If 95% of the cases we don't want to have timestamps in consolidated fragments then it'd make sense to change the default value to false. cc @ihnorton

Consolidation is not only a question of data retention, but also data management. There might be several reasons that DBAs would prefer to have one larger consolidated fragment versus a larger number of non-consolidated smaller fragments. I am not a DBA so I can't really enumerate them.

From the perspective of a query engine there is a difference:

If you have a query which wants half of your fragments, and you are not consolidated, then you have to merge coordinates from all of the fragments.

If you have a query which wants half of your fragments, and you are consolidated, then instead you have to de-duplicate a single stream on the max timestamp per coordinate.

I would expect merge to be a lot more resource-intensive than a single-stream de-duplicate. Imagine the effort required to parallelize them, for example.

If 95% of the cases we don't want to have timestamps in consolidated fragments then it'd make sense to change the default value to false

I'm a bit leery of this. The upshot of false is performance; the downside is what a customer doing the wrong thing would perceive as data loss.

[rroelke]

If you consolidate with timestamps, and then decide you want to purge old data, does it work to re-run consolidation in the "purge" mode?

[ihnorton]

The underlying goal here is to optimize an array for efficient reads with time-traveling across the fragment history, and that remains a requirement which we will continue to support. The implementation may evolve or change to better realize the usage requirements.