feat: fall back to uncompressed copy on unbacked sparse-index metadata - #199
Merged
Conversation
arajkumar
reviewed
Jul 28, 2026
A chunk compressed under a pre-2.28 TimescaleDB format carries `minmax` sparse-index metadata only. When such a chunk is backfilled into a target whose hypertable was configured under TS 2.28, `create_compressed_chunk` copies the target's hypertable-level index (which includes `firstlast`) onto the new chunk. That references physical `_ts_meta_*` columns the copied data lacks, corrupting the chunk's catalog: on PG16 targets every query over the chunk then fails with `cache lookup failed for attribute 0` (issue #195). Reconciling the catalog after the fact isn't viable on Tiger Cloud, where `tsdbadmin` lacks UPDATE on `compression_settings`, and no GUC suppresses the copy. Instead, detect the mismatch during `stage` (when the target chunk structure is pre-created) and fail loudly with remediation guidance: decompress and recompress the affected source chunks so their physical layout matches the current format before copying. The comparison logic is unit-tested; an integration test reproduces the end-to-end scenario on a 2.28 target with an old-format source chunk.
Failing the backfill outright leaves users of an older source with no way forward: rebuilding the source chunks only yields a compatible layout once the source extension is itself on 2.28. These chunks can still be copied correctly, just not in compressed form. Detect the mismatch by comparing the target hypertable's declared sparse-index entries against the physical `_ts_meta_*` columns the source compressed chunk actually has, then copy the affected chunks uncompressed and let the target compress them in its own format. `stage` reports them up front and skips pre-creating their target compressed chunk structure. `copy` decides this before the completion-filter branch, which recompresses the target chunk only when it was already compressed: otherwise the chunk holding the `--until` boundary would stay uncompressed forever. A target compressed chunk pre-created by an older binary is diagnosed before decompression, so PG16 reports what happened instead of surfacing `cache lookup failed for attribute 0`. Detection fails closed: an unrecognized `compression_settings.index` shape, a malformed entry and a missing settings row are errors, and an unknown entry type counts as unbacked. The integration tests assert directly that every `firstlast` entry in the target's chunk-level `compression_settings.index` has both backing columns in `pg_attribute`. Row and chunk counts can't stand in for this: PG17 tolerates the dangling metadata, so every count assertion passes there even when the corrupt path ran.
alejandrodnm
force-pushed
the
adn/195-sparse-index-metadata-guard
branch
from
August 7, 2026 07:27
f23085f to
d039830
Compare
arajkumar
approved these changes
Aug 10, 2026
Member
|
/td-code-review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Fixes #195.
Backfilling a chunk that was compressed under a pre-2.28 TimescaleDB format
into a target whose hypertable was configured under TS 2.28 corrupts the target
chunk's catalog. The old chunk carries
minmaxsparse-index metadata only, butcreate_compressed_chunkcopies the target hypertable-level index (whichincludes
firstlast) onto the new chunk. The copiedfirstlastentryreferences physical
_ts_meta_*columns the data lacks, so on PG16 targetsevery query over the chunk fails with
cache lookup failed for attribute 0; onPG17 the planner silently tolerates the dangling metadata.
Reconciling the catalog after the fact isn't an option on Tiger Cloud
(
tsdbadminhas noUPDATEoncompression_settings), and there's no GUC tosuppress the copy.
Failing the backfill outright would leave users of an older source with no way
forward: rebuilding the source chunks only produces a compatible layout once the
source extension is itself on 2.28. These chunks can still be copied correctly,
just not in compressed form, so the tool takes that path instead of refusing to
migrate.
What
target compress them in its own format. Detection compares the target
hypertable's declared sparse-index entries against the physical
_ts_meta_*columns the source compressed chunk actually has, so it tests catalog against
layout, which is where the drift lives.
stage, before any data moves, and skippre-creating their target compressed chunk structure. The warning names the
chunk count, the unbacked entries, the issue, and the remediation for copying
them compressed instead.
copy, which recompressesthe target chunk only when it was already compressed. Otherwise the chunk
holding the
--untilboundary would stay uncompressed forever.decompressing it, so PG16 reports what actually happened instead of surfacing
cache lookup failed for attribute 0.compression_settings.indexshape, amalformed entry and a missing settings row are all errors, and an entry type
the tool doesn't know is treated as unbacked.
hypertable_sparse_index_metadata).Tests
comparison, including the fail-closed cases.
copy_falls_back_to_uncompressed_on_unbacked_sparse_index_metadatareproducesthe scenario end-to-end: compress the source under TS 2.27.2, upgrade the
source to 2.28, configure the target natively under 2.28, then assert the
stage warning, the copy-time fallback, 744 rows in 5 compressed chunks, and a
clean
verify.filtered_copy_still_recompresses_on_unbacked_sparse_index_metadatastageswith an
--untilinside the last chunk and asserts the target still ends upwith 5 compressed chunks.
firstlastentry in the target's chunk-levelcompression_settings.indexhas both backing columns inpg_attribute. Rowand chunk counts can't stand in for this: PG17 tolerates the dangling
metadata, so every count assertion passes there even when the corrupt path
ran.
SKIPPED <test>: requires TimescaleDB 2.28instead of reporting a silent success on the 2.27, 2.29 and nightly rows.