Skip to content

feat: fall back to uncompressed copy on unbacked sparse-index metadata - #199

Merged
alejandrodnm merged 2 commits into
mainfrom
adn/195-sparse-index-metadata-guard
Aug 26, 2026
Merged

feat: fall back to uncompressed copy on unbacked sparse-index metadata#199
alejandrodnm merged 2 commits into
mainfrom
adn/195-sparse-index-metadata-guard

Conversation

@alejandrodnm

@alejandrodnm alejandrodnm commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Why

Fixes #195.

Backfilling a chunk that was compressed under a pre-2.28 TimescaleDB format
into a target whose hypertable was configured under TS 2.28 corrupts the target
chunk's catalog. The old chunk carries minmax sparse-index metadata only, but
create_compressed_chunk copies the target hypertable-level index (which
includes firstlast) onto the new chunk. The copied firstlast entry
references physical _ts_meta_* columns the data lacks, so on PG16 targets
every query over the chunk fails with cache lookup failed for attribute 0; on
PG17 the planner silently tolerates the dangling metadata.

Reconciling the catalog after the fact isn't an option on Tiger Cloud
(tsdbadmin has no UPDATE on compression_settings), and there's no GUC to
suppress the copy.

Failing the backfill outright would leave users of an older source with no way
forward: rebuilding the source chunks only produces a compatible layout once the
source extension is itself on 2.28. These chunks can still be copied correctly,
just not in compressed form, so the tool takes that path instead of refusing to
migrate.

What

  • Detect the mismatch and copy the affected chunks uncompressed, letting the
    target compress them in its own format. Detection compares the target
    hypertable's declared sparse-index entries against the physical _ts_meta_*
    columns the source compressed chunk actually has, so it tests catalog against
    layout, which is where the drift lives.
  • Report the affected chunks during stage, before any data moves, and skip
    pre-creating their target compressed chunk structure. The warning names the
    chunk count, the unbacked entries, the issue, and the remediation for copying
    them compressed instead.
  • Decide this before the completion-filter branch in copy, which recompresses
    the target chunk only when it was already compressed. Otherwise the chunk
    holding the --until boundary would stay uncompressed forever.
  • Diagnose a target compressed chunk pre-created by an older binary before
    decompressing it, so PG16 reports what actually happened instead of surfacing
    cache lookup failed for attribute 0.
  • Fail closed throughout: an unrecognized compression_settings.index shape, a
    malformed entry and a missing settings row are all errors, and an entry type
    the tool doesn't know is treated as unbacked.
  • Gate the check on target TS >= 2.28 (hypertable_sparse_index_metadata).

Tests

  • Unit tests for the parsing, the backing-column mapping and the unbacked
    comparison, including the fail-closed cases.
  • copy_falls_back_to_uncompressed_on_unbacked_sparse_index_metadata reproduces
    the scenario end-to-end: compress the source under TS 2.27.2, upgrade the
    source to 2.28, configure the target natively under 2.28, then assert the
    stage warning, the copy-time fallback, 744 rows in 5 compressed chunks, and a
    clean verify.
  • filtered_copy_still_recompresses_on_unbacked_sparse_index_metadata stages
    with an --until inside the last chunk and asserts the target still ends up
    with 5 compressed chunks.
  • Both assert directly that every firstlast entry in the target's chunk-level
    compression_settings.index has both backing columns in pg_attribute. Row
    and chunk counts can't stand in for this: PG17 tolerates the dangling
    metadata, so every count assertion passes there even when the corrupt path
    ran.
  • Version-gated skips now print SKIPPED <test>: requires TimescaleDB 2.28
    instead of reporting a silent success on the 2.27, 2.29 and nightly rows.

Comment thread src/execute.rs Outdated
@alejandrodnm alejandrodnm changed the title feat: guard against unbacked sparse-index metadata feat: fall back to uncompressed copy on unbacked sparse-index metadata Aug 7, 2026
A chunk compressed under a pre-2.28 TimescaleDB format carries `minmax`
sparse-index metadata only. When such a chunk is backfilled into a target
whose hypertable was configured under TS 2.28, `create_compressed_chunk`
copies the target's hypertable-level index (which includes `firstlast`)
onto the new chunk. That references physical `_ts_meta_*` columns the
copied data lacks, corrupting the chunk's catalog: on PG16 targets every
query over the chunk then fails with `cache lookup failed for attribute 0`
(issue #195).

Reconciling the catalog after the fact isn't viable on Tiger Cloud, where
`tsdbadmin` lacks UPDATE on `compression_settings`, and no GUC suppresses
the copy. Instead, detect the mismatch during `stage` (when the target
chunk structure is pre-created) and fail loudly with remediation guidance:
decompress and recompress the affected source chunks so their physical
layout matches the current format before copying.

The comparison logic is unit-tested; an integration test reproduces the
end-to-end scenario on a 2.28 target with an old-format source chunk.
Failing the backfill outright leaves users of an older source with no way
forward: rebuilding the source chunks only yields a compatible layout once the
source extension is itself on 2.28. These chunks can still be copied correctly,
just not in compressed form.

Detect the mismatch by comparing the target hypertable's declared sparse-index
entries against the physical `_ts_meta_*` columns the source compressed chunk
actually has, then copy the affected chunks uncompressed and let the target
compress them in its own format. `stage` reports them up front and skips
pre-creating their target compressed chunk structure.

`copy` decides this before the completion-filter branch, which recompresses the
target chunk only when it was already compressed: otherwise the chunk holding
the `--until` boundary would stay uncompressed forever. A target compressed
chunk pre-created by an older binary is diagnosed before decompression, so PG16
reports what happened instead of surfacing `cache lookup failed for attribute
0`.

Detection fails closed: an unrecognized `compression_settings.index` shape, a
malformed entry and a missing settings row are errors, and an unknown entry type
counts as unbacked.

The integration tests assert directly that every `firstlast` entry in the
target's chunk-level `compression_settings.index` has both backing columns in
`pg_attribute`. Row and chunk counts can't stand in for this: PG17 tolerates the
dangling metadata, so every count assertion passes there even when the corrupt
path ran.
@alejandrodnm
alejandrodnm force-pushed the adn/195-sparse-index-metadata-guard branch from f23085f to d039830 Compare August 7, 2026 07:27
@alejandrodnm
alejandrodnm requested review from a team and arajkumar August 10, 2026 07:42
@arajkumar

Copy link
Copy Markdown
Member

/td-code-review

@alejandrodnm
alejandrodnm merged commit d97aaa6 into main Aug 26, 2026
13 checks passed
@alejandrodnm
alejandrodnm deleted the adn/195-sparse-index-metadata-guard branch August 26, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Backfilled compressed chunks get 'firstlast' sparse-index metadata without backing columns — all queries fail on PG16 targets (TS 2.28.2)

2 participants