Skip to content

Support RETURNING clause with direct compress - #10280

Merged
svenklemm merged 1 commit into
mainfrom
sven/dc_returning
Jul 18, 2026
Merged

Support RETURNING clause with direct compress#10280
svenklemm merged 1 commit into
mainfrom
sven/dc_returning

Conversation

@svenklemm

@svenklemm svenklemm commented Jul 16, 2026

Copy link
Copy Markdown
Member

The direct compress insert path added each tuple to the compressor and
moved on to the next row without ever evaluating the RETURNING clause,
so an INSERT with RETURNING returned no rows even though the data was
inserted. Project and return the RETURNING result for each inserted row
in the direct compress path, matching the normal insert path.

Fixes #10243

@github-actions
github-actions Bot requested review from akuzm and antekresic July 16, 2026 20:25
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown

@antekresic, @akuzm: please review this pull request.

Powered by pull-review

@svenklemm svenklemm added this to the v2.29.0 milestone Jul 16, 2026
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@svenklemm
Support RETURNING clause with direct compress
42cc89d 
The direct compress insert path added each tuple to the compressor and
moved on to the next row without ever evaluating the RETURNING clause,
so an INSERT with RETURNING returned no rows even though the data was
inserted. Project and return the RETURNING result for each inserted row
in the direct compress path, matching the normal insert path.

Fixes #10243
@svenklemm
svenklemm force-pushed the sven/dc_returning branch from b35d5d1 to 42cc89d Compare July 18, 2026 07:32
@svenklemm
svenklemm merged commit c9baf4b into main Jul 18, 2026
61 checks passed
@svenklemm
svenklemm deleted the sven/dc_returning branch July 18, 2026 07:48
@timescale-automation

timescale-automation commented Jul 18, 2026

Copy link
Copy Markdown
Member

Automated backport to 2.28.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.28.x
You are currently cherry-picking commit c9baf4be7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .unreleased/fix_direct_compress_returning

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   src/nodes/modify_hypertable_exec.c
	both modified:   tsl/test/expected/direct_compress_insert.out
	both modified:   tsl/test/sql/direct_compress_insert.sql

Job log

@timescale-automation timescale-automation added the auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) label Jul 18, 2026
@surister surister mentioned this pull request Jul 21, 2026
Merged
@philkra philkra removed the auto-backport-not-done Automated backport of this PR has failed non-retriably (e.g. conflicts) label Jul 25, 2026
surister pushed a commit that referenced this pull request Jul 28, 2026
@timescale-automation
Release 2.29.0
0ba495e 
## 2.29.0 (2026-07-28)

This release contains performance improvements and bug fixes since the
2.28.3 release. We recommend that you upgrade at the next available
opportunity.

**Release Highlights**
* **Chunk exclusion for DML operations** drastically improves the
performance of `UPDATE` and `DELETE` statements on hypertables. By
acquiring exclusive locks only on the specific chunks being modified
rather than the entire hypertable, this enhancement eliminates massive
lock contention and keeps high-concurrency workloads running smoothly
without unnecessary slowdowns.
* Intelligent **row-by-row decompression** enables the query planner to
decompress data row-by-row rather than in large batches when an
operation prioritizes a fast initial response (such as queries with
`LIMIT` clauses). This dramatically reduces memory overhead and query
latency, ensuring lightning-fast performance when you only need to
retrieve a small subset of records from your compressed hypertables.

**Important: PostgreSQL 15 Support Removed**
TimescaleDB 2.29.0 removes support for PostgreSQL 15. This release
supports PostgreSQL 16, 17, and 18. If you are still running PostgreSQL
15, upgrade PostgreSQL before upgrading to TimescaleDB 2.29.0.

**Backward-Incompatible Changes**
* [#10041](#10041) Remove
support for PostgreSQL 15

**Features**
* [#9315](#9315) Speed up
`DML` operations on hypertables by using the optimized TimescaleDB
hypertable expansion code instead of the generic PostgreSQL inheritance
hierarchy expansion
* [#9534](#9534) Speed up
expression evaluation in the columnar pipeline by caching common
subexpressions
* [#9684](#9684) Add
`_timescaledb_functions.decompress_batch()` SQL function
* [#9732](#9732) Speed up
some queries with small `LIMIT` by switching to row-by-row query
execution pipeline
* [#9917](#9917) Decompress
less data in `DML` on compressed hypertables by accounting for prepared
statement parameters
* [#9957](#9957) Add
`compact_chunk()` function
* [#10048](#10048) Support
concurrent refresh policies on hierarchical continuous aggregates
* [#10081](#10081) Add
`samplerate` argument to
`_timescaledb_functions.estimate_uncompressed_size()`
* [#10100](#10100) Skip
classifying compressed relations to speed up planning
* [#10118](#10118) Don't
track compressed relations as separate chunk
* [#10119](#10119) Reduce
memory usage of `INSERT` queries using direct compress and spanning
multiple chunks
* [#10163](#10163) Add a
compaction policy for unordered chunks
* [#10204](#10204) Don't
create separate hypertable catalog entry for hypertables with
compression
* [#10217](#10217) Initial
placeholder version of granular refresh API
* [#10225](#10225) Add
`config_merge` parameter to `alter_job()` for merging `jsonb` into the
existing job configuration
* [#10226](#10226) Add
`recompress_unordered` columnstore policy option
* [#10231](#10231) Use
`regclass` for storing relation reference in chunk table
* [#10237](#10237) Add
helper functions for decoding hypertable status
* [#10240](#10240) Add the
`tsdb.direct_compress` storage parameter that allows enabling direct
compress for a given hypertable independent of global settings
* [#10266](#10266) Add
`max_batches` to `compact_chunk()`
* [#10299](#10299) Add
`continuous_aggs_tenant_tracking` and `hypertable_cagg_settings`
catalogs

**Bugfixes**
* [#10013](#10013) Make
ownership error messages on continuous aggregates consistent
* [#10052](#10052) Result
of `MIN` / `MAX` aggregate functions in columnar aggregation pipeline
possibly inconsistent with plain PostgreSQL result
* [#10071](#10071) Prune
the real-time branch of hierarchical continuous aggregates at any
nesting depth
* [#10143](#10143) Fix
division by zero when planning `time_bucket` with zero width
* [#10199](#10199) Fix
`initial_start` handling in `build_job_info`
* [#10213](#10213) Cache
sort pathkeys per hypertable
* [#10221](#10221) Fix
incremental refresh skipping the last bucket
* [#10278](#10278) Drop
`job_errors` view in `bgw_job_stat_history` migration
* [#10280](#10280)
`RETURNING` clause returned no rows for `INSERT` using direct compress
* [#10281](#10281) Disable
direct compress when the destination table has an exclusion constraint
so the constraint is still enforced
* [#10282](#10282) Only
count directly compressed rows toward the command tag when the `INSERT`
sets it
* [#10286](#10286)
Propagate `VACUUM` on a chunk to the compressed relation when running on
the chunk directly
* [#10302](#10302) Fix
useless-join removal and self-join elimination for hypertables
* [#10313](#10313) Allow
running `ALTER EXTENSION timescaledb UPDATE` inside a transaction block
* [#10315](#10315) Fix
overlap detection with running max
* [#10324](#10324) Fix
stale index entries after `rebuild_sparse_index()` on compressed chunks

**GUCs**
* `timescaledb.enable_hypertable_expansion_for_dml`: allow using the
optimized TimescaleDB hypertable expansion code for `UPDATE` and
`DELETE` instead of the generic PostgreSQL inheritance hierarchy
expansion. On by default.

**Thanks**
* @FrancescEthon and @ManuelEthon for reporting the issue
* @h0rn3t for reporting a problem with `VACUUM` not propagating to the
compressed relation
* @MaximeEthon for reporting an issue with prepared statement parameters
in DML decompression
* @proddata for reporting a problem when upgrading from 2.15.3 to 2.28.2
* @tureba for reporting and fixing stale sparse-index entries after
rebuild
* @viniciusrsouza for reporting an issue with hierarchical continuous
aggregates
svenklemm pushed a commit that referenced this pull request Jul 28, 2026
@timescale-automation @svenklemm
Release 2.29.0
5be1737 
This release contains performance improvements and bug fixes since the
2.28.3 release. We recommend that you upgrade at the next available
opportunity.

**Release Highlights**
* **Chunk exclusion for DML operations** drastically improves the
performance of `UPDATE` and `DELETE` statements on hypertables. By
acquiring exclusive locks only on the specific chunks being modified
rather than the entire hypertable, this enhancement eliminates massive
lock contention and keeps high-concurrency workloads running smoothly
without unnecessary slowdowns.
* Intelligent **row-by-row decompression** enables the query planner to
decompress data row-by-row rather than in large batches when an
operation prioritizes a fast initial response (such as queries with
`LIMIT` clauses). This dramatically reduces memory overhead and query
latency, ensuring lightning-fast performance when you only need to
retrieve a small subset of records from your compressed hypertables.

**Important: PostgreSQL 15 Support Removed**
TimescaleDB 2.29.0 removes support for PostgreSQL 15. This release
supports PostgreSQL 16, 17, and 18. If you are still running PostgreSQL
15, upgrade PostgreSQL before upgrading to TimescaleDB 2.29.0.

**Backward-Incompatible Changes**
* [#10041](#10041) Remove
support for PostgreSQL 15

**Features**
* [#9315](#9315) Speed up
`DML` operations on hypertables by using the optimized TimescaleDB
hypertable expansion code instead of the generic PostgreSQL inheritance
hierarchy expansion
* [#9534](#9534) Speed up
expression evaluation in the columnar pipeline by caching common
subexpressions
* [#9684](#9684) Add
`_timescaledb_functions.decompress_batch()` SQL function
* [#9732](#9732) Speed up
some queries with small `LIMIT` by switching to row-by-row query
execution pipeline
* [#9917](#9917) Decompress
less data in `DML` on compressed hypertables by accounting for prepared
statement parameters
* [#9957](#9957) Add
`compact_chunk()` function
* [#10048](#10048) Support
concurrent refresh policies on hierarchical continuous aggregates
* [#10081](#10081) Add
`samplerate` argument to
`_timescaledb_functions.estimate_uncompressed_size()`
* [#10100](#10100) Skip
classifying compressed relations to speed up planning
* [#10118](#10118) Don't
track compressed relations as separate chunk
* [#10119](#10119) Reduce
memory usage of `INSERT` queries using direct compress and spanning
multiple chunks
* [#10163](#10163) Add a
compaction policy for unordered chunks
* [#10204](#10204) Don't
create separate hypertable catalog entry for hypertables with
compression
* [#10217](#10217) Initial
placeholder version of granular refresh API
* [#10225](#10225) Add
`config_merge` parameter to `alter_job()` for merging `jsonb` into the
existing job configuration
* [#10226](#10226) Add
`recompress_unordered` columnstore policy option
* [#10231](#10231) Use
`regclass` for storing relation reference in chunk table
* [#10237](#10237) Add
helper functions for decoding hypertable status
* [#10240](#10240) Add the
`tsdb.direct_compress` storage parameter that allows enabling direct
compress for a given hypertable independent of global settings
* [#10266](#10266) Add
`max_batches` to `compact_chunk()`
* [#10299](#10299) Add
`continuous_aggs_tenant_tracking` and `hypertable_cagg_settings`
catalogs

**Bugfixes**
* [#10013](#10013) Make
ownership error messages on continuous aggregates consistent
* [#10052](#10052) Result
of `MIN` / `MAX` aggregate functions in columnar aggregation pipeline
possibly inconsistent with plain PostgreSQL result
* [#10071](#10071) Prune
the real-time branch of hierarchical continuous aggregates at any
nesting depth
* [#10143](#10143) Fix
division by zero when planning `time_bucket` with zero width
* [#10199](#10199) Fix
`initial_start` handling in `build_job_info`
* [#10213](#10213) Cache
sort pathkeys per hypertable
* [#10221](#10221) Fix
incremental refresh skipping the last bucket
* [#10278](#10278) Drop
`job_errors` view in `bgw_job_stat_history` migration
* [#10280](#10280)
`RETURNING` clause returned no rows for `INSERT` using direct compress
* [#10281](#10281) Disable
direct compress when the destination table has an exclusion constraint
so the constraint is still enforced
* [#10282](#10282) Only
count directly compressed rows toward the command tag when the `INSERT`
sets it
* [#10286](#10286)
Propagate `VACUUM` on a chunk to the compressed relation when running on
the chunk directly
* [#10302](#10302) Fix
useless-join removal and self-join elimination for hypertables
* [#10313](#10313) Allow
running `ALTER EXTENSION timescaledb UPDATE` inside a transaction block
* [#10315](#10315) Fix
overlap detection with running max
* [#10324](#10324) Fix
stale index entries after `rebuild_sparse_index()` on compressed chunks

**GUCs**
* `timescaledb.enable_hypertable_expansion_for_dml`: allow using the
optimized TimescaleDB hypertable expansion code for `UPDATE` and
`DELETE` instead of the generic PostgreSQL inheritance hierarchy
expansion. On by default.

**Thanks**
* @FrancescEthon and @ManuelEthon for reporting an issue with incremental refresh skipping the last bucket
* @h0rn3t for reporting a problem with `VACUUM` not propagating to the
* @igor2x for reporting an issue with locking during DML statements on hypertables
compressed relation
* @MaximeEthon for reporting an issue with prepared statement parameters
in DML decompression
* @proddata for reporting a problem when upgrading from 2.15.3 to 2.28.2
* @tureba for reporting and fixing stale sparse-index entries after
rebuild
* @viniciusrsouza for reporting an issue with hierarchical continuous
aggregates
surister pushed a commit to surister/timescaledb that referenced this pull request Jul 28, 2026
@timescale-automation @surister
Release 2.29.0
ccedf24 
This release contains performance improvements and bug fixes since the
2.28.3 release. We recommend that you upgrade at the next available
opportunity.

**Release Highlights**
* **Chunk exclusion for DML operations** drastically improves the
performance of `UPDATE` and `DELETE` statements on hypertables. By
acquiring exclusive locks only on the specific chunks being modified
rather than the entire hypertable, this enhancement eliminates massive
lock contention and keeps high-concurrency workloads running smoothly
without unnecessary slowdowns.
* Intelligent **row-by-row decompression** enables the query planner to
decompress data row-by-row rather than in large batches when an
operation prioritizes a fast initial response (such as queries with
`LIMIT` clauses). This dramatically reduces memory overhead and query
latency, ensuring lightning-fast performance when you only need to
retrieve a small subset of records from your compressed hypertables.

**Important: PostgreSQL 15 Support Removed**
TimescaleDB 2.29.0 removes support for PostgreSQL 15. This release
supports PostgreSQL 16, 17, and 18. If you are still running PostgreSQL
15, upgrade PostgreSQL before upgrading to TimescaleDB 2.29.0.

**Backward-Incompatible Changes**
* [timescale#10041](timescale#10041) Remove
support for PostgreSQL 15

**Features**
* [timescale#9315](timescale#9315) Speed up
`DML` operations on hypertables by using the optimized TimescaleDB
hypertable expansion code instead of the generic PostgreSQL inheritance
hierarchy expansion
* [timescale#9534](timescale#9534) Speed up
expression evaluation in the columnar pipeline by caching common
subexpressions
* [timescale#9684](timescale#9684) Add
`_timescaledb_functions.decompress_batch()` SQL function
* [timescale#9732](timescale#9732) Speed up
some queries with small `LIMIT` by switching to row-by-row query
execution pipeline
* [timescale#9917](timescale#9917) Decompress
less data in `DML` on compressed hypertables by accounting for prepared
statement parameters
* [timescale#9957](timescale#9957) Add
`compact_chunk()` function
* [timescale#10048](timescale#10048) Support
concurrent refresh policies on hierarchical continuous aggregates
* [timescale#10081](timescale#10081) Add
`samplerate` argument to
`_timescaledb_functions.estimate_uncompressed_size()`
* [timescale#10100](timescale#10100) Skip
classifying compressed relations to speed up planning
* [timescale#10118](timescale#10118) Don't
track compressed relations as separate chunk
* [timescale#10119](timescale#10119) Reduce
memory usage of `INSERT` queries using direct compress and spanning
multiple chunks
* [timescale#10163](timescale#10163) Add a
compaction policy for unordered chunks
* [timescale#10204](timescale#10204) Don't
create separate hypertable catalog entry for hypertables with
compression
* [timescale#10217](timescale#10217) Initial
placeholder version of granular refresh API
* [timescale#10225](timescale#10225) Add
`config_merge` parameter to `alter_job()` for merging `jsonb` into the
existing job configuration
* [timescale#10226](timescale#10226) Add
`recompress_unordered` columnstore policy option
* [timescale#10231](timescale#10231) Use
`regclass` for storing relation reference in chunk table
* [timescale#10237](timescale#10237) Add
helper functions for decoding hypertable status
* [timescale#10240](timescale#10240) Add the
`tsdb.direct_compress` storage parameter that allows enabling direct
compress for a given hypertable independent of global settings
* [timescale#10266](timescale#10266) Add
`max_batches` to `compact_chunk()`
* [timescale#10299](timescale#10299) Add
`continuous_aggs_tenant_tracking` and `hypertable_cagg_settings`
catalogs

**Bugfixes**
* [timescale#10013](timescale#10013) Make
ownership error messages on continuous aggregates consistent
* [timescale#10052](timescale#10052) Result
of `MIN` / `MAX` aggregate functions in columnar aggregation pipeline
possibly inconsistent with plain PostgreSQL result
* [timescale#10071](timescale#10071) Prune
the real-time branch of hierarchical continuous aggregates at any
nesting depth
* [timescale#10143](timescale#10143) Fix
division by zero when planning `time_bucket` with zero width
* [timescale#10199](timescale#10199) Fix
`initial_start` handling in `build_job_info`
* [timescale#10213](timescale#10213) Cache
sort pathkeys per hypertable
* [timescale#10221](timescale#10221) Fix
incremental refresh skipping the last bucket
* [timescale#10278](timescale#10278) Drop
`job_errors` view in `bgw_job_stat_history` migration
* [timescale#10280](timescale#10280)
`RETURNING` clause returned no rows for `INSERT` using direct compress
* [timescale#10281](timescale#10281) Disable
direct compress when the destination table has an exclusion constraint
so the constraint is still enforced
* [timescale#10282](timescale#10282) Only
count directly compressed rows toward the command tag when the `INSERT`
sets it
* [timescale#10286](timescale#10286)
Propagate `VACUUM` on a chunk to the compressed relation when running on
the chunk directly
* [timescale#10302](timescale#10302) Fix
useless-join removal and self-join elimination for hypertables
* [timescale#10313](timescale#10313) Allow
running `ALTER EXTENSION timescaledb UPDATE` inside a transaction block
* [timescale#10315](timescale#10315) Fix
overlap detection with running max
* [timescale#10324](timescale#10324) Fix
stale index entries after `rebuild_sparse_index()` on compressed chunks

**GUCs**
* `timescaledb.enable_hypertable_expansion_for_dml`: allow using the
optimized TimescaleDB hypertable expansion code for `UPDATE` and
`DELETE` instead of the generic PostgreSQL inheritance hierarchy
expansion. On by default.

**Thanks**
* @FrancescEthon and @ManuelEthon for reporting an issue with incremental refresh skipping the last bucket
* @h0rn3t for reporting a problem with `VACUUM` not propagating to the
* @igor2x for reporting an issue with locking during DML statements on hypertables
compressed relation
* @MaximeEthon for reporting an issue with prepared statement parameters
in DML decompression
* @proddata for reporting a problem when upgrading from 2.15.3 to 2.28.2
* @tureba for reporting and fixing stale sparse-index entries after
rebuild
* @viniciusrsouza for reporting an issue with hierarchical continuous
aggregates
@timescale-automation timescale-automation added the released-2.29.0 Released in 2.29.0 label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dbeck dbeck dbeck approved these changes
@Poroma-Banerjee Poroma-Banerjee Poroma-Banerjee approved these changes
@akuzm akuzm Awaiting requested review from akuzm
@antekresic antekresic Awaiting requested review from antekresic

Assignees

@svenklemm svenklemm

Labels

released-2.29.0 Released in 2.29.0

Projects

None yet

Milestone

v2.29.0

Development

Successfully merging this pull request may close these issues.

[Bug]: RETURNING clause doesn't work with direct compress

5 participants

@svenklemm @timescale-automation @dbeck @Poroma-Banerjee @philkra