Document direct compress table option and compaction policy (2.29.0)

astro.config.ts

@@ -1375,6 +1375,8 @@ export default defineConfig({
                       items: [
                         { label: "add_columnstore_policy()", link: "/reference/timescaledb/hypercore/add_columnstore_policy" },
                         { label: "remove_columnstore_policy()", link: "/reference/timescaledb/hypercore/remove_columnstore_policy" },
+                        { label: "add_compaction_policy()", link: "/reference/timescaledb/hypercore/add_compaction_policy" },
+                        { label: "remove_compaction_policy()", link: "/reference/timescaledb/hypercore/remove_compaction_policy" },
                       ],
                     },
                     {

src/content/docs/reference/timescaledb/hypercore/add_compaction_policy.mdx

@@ -0,0 +1,84 @@
+---
+title: add_compaction_policy()
+description: Set a policy to automatically compact unordered chunks in the columnstore
+products: [cloud, self_hosted]
+keywords: [columnstore, hypercore, policies, compaction, direct compress]
+---
+
+import * as C from "@constants";
+
+import SinceRelease from '@components/SinceRelease.astro';
+
+import { Callout } from '@stainless-api/docs/components';
+
+<SinceRelease version="2.29.0" product="tsdb" />
+
+Create a [job](/reference/timescaledb/jobs-automation/add_job) that automatically compacts unordered {C.CHUNK}s in the
+{C.COLUMNSTORE}. Compaction combines overlapping batches within a {C.CHUNK} so queries no longer need an extra sort step
+to restore order.
+
+<Callout variant="note">
+
+You usually don't need to add this policy yourself. [Direct compress](/build/data-management/write-data/insert#improve-performance-with-direct-compress)
+creates it for you, and it is the main reason {C.CHUNK}s end up unordered. Use `add_compaction_policy` when you want to
+tune the schedule or bound how much work each run does. Direct compress is a tech preview, so this policy is too.
+
+</Callout>
+
+The policy only processes {C.CHUNK}s that are fully in the {C.COLUMNSTORE} and marked as unordered. It skips partial
+{C.CHUNK}s, which the [{C.COLUMNSTORE} policy](/reference/timescaledb/hypercore/add_columnstore_policy) handles, and frozen {C.CHUNK}s. If your {C.HYPERTABLE}
+doesn't use direct compress, it is unlikely to have unordered {C.CHUNK}s and this policy has nothing to do.
+
+To view the policies that you set or the policies that already exist, see [informational views](/reference/timescaledb/informational-views/jobs).
+
+## Samples
+
+- **Add a compaction policy with the default 5 minute schedule**:
+
+  ```sql
+  SELECT add_compaction_policy('metrics');
+  ```
+
+- **Run less often, bound the work per run, and skip {C.CHUNK}s still being written to**:
+
+  ```sql
+  SELECT add_compaction_policy('metrics',
+     schedule_interval => INTERVAL '15 minutes',
+     max_chunks => 10,
+     max_batches => 500,
+     inactive_for => INTERVAL '30 minutes');
+  ```
+
+## Arguments
+
+The syntax is:
+
+```sql
+SELECT add_compaction_policy(
+    hypertable = '<hypertable_name>',
+    if_not_exists = true | false,
+    schedule_interval = <interval>,
+    initial_start = <timestamptz>,
+    timezone = '<timezone>',
+    max_chunks = <integer>,
+    max_batches = <integer>,
+    inactive_for = <interval>
+);
+```
+
+| Name                | Type        | Default      | Required | Description                                                                                                                                                                                     |
+|---------------------|-------------|--------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `hypertable`        | REGCLASS    | -            | ✔        | Name of the {C.HYPERTABLE} to run this [job](/reference/timescaledb/jobs-automation/add_job) on.                                                                                                                                   |
+| `if_not_exists`     | BOOLEAN     | `false`      | ✖        | Set to `true` so this job fails with a warning rather than an error if a compaction policy already exists on `hypertable`.                                                                       |
+| `schedule_interval` | INTERVAL    | `5 minutes`  | ✖        | Set the interval between the finish time of the last execution of this policy and the next start. When direct compress creates this policy, it uses `1 minute` instead.                          |
+| `initial_start`     | TIMESTAMPTZ | `NULL`       | ✖        | Set the time this job is first run.                                                                                                                                                             |
+| `timezone`          | TEXT        | `NULL`       | ✖        | Set to a valid time zone to mitigate DST shifting. If `initial_start` is set, subsequent executions of this policy are aligned on `initial_start`.                                               |
+| `max_chunks`        | INTEGER     | `NULL`       | ✖        | Set the maximum number of {C.CHUNK}s to process in a single run, including {C.CHUNK}s that fail. Leave unset to process every eligible {C.CHUNK}.                                                            |
+| `max_batches`       | INTEGER     | `NULL`       | ✖        | Set the maximum number of batches to combine in each {C.CHUNK}. Leave unset for no limit.                                                                                                             |
+| `inactive_for`      | INTERVAL    | `NULL`       | ✖        | Only compact {C.CHUNK}s that have not been written to for this interval. Leave unset to compact every eligible {C.CHUNK} regardless of when it was last written to.                                        |
+
+## Returns
+
+|Column|Type|Description|
+|-|-|-|
+|`job_id`|INTEGER|{C.TIMESCALE_DB} background job ID created to implement this policy|

src/content/docs/reference/timescaledb/hypercore/alter_table.mdx

@@ -80,6 +80,7 @@ ALTER TABLE <table_name> SET (timescaledb.enable_columnstore,
    timescaledb.compress_segmentby = '<column_name> [, ...]',
    timescaledb.sparse_index = '<index>(<column_name>), <index>(<column_name>)',
    timescaledb.compress_chunk_time_interval='interval',
+   timescaledb.direct_compress = true | false,
    ALTER <column name> SET NOT NULL,
    ADD CONSTRAINT <constraint_name> UNIQUE (<column name>, ... )
 );
@@ -95,6 +96,7 @@ ALTER TABLE <table_name> SET (timescaledb.enable_columnstore,
 |`timescaledb.sparse_index`| TEXT    | {C.TIMESCALE_DB} evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization.                                                         | ✖        | Configure the sparse indexes for {C.COLUMNSTORE} {C.CHUNK}s. Requires setting `timescaledb.compress_orderby`. Aliases: `timescaledb.index`, `timescaledb.compress_index`, `timescaledb.compress_sparse_index`. Supported index types are `bloom(<column>, …)` for equality filters and `minmax(<column>)` for range filters; use a comma-separated list to set multiple. For when each is best, configuration patterns, and restrictions, see [Sparse indexes on the columnstore](/build/performance-optimization/indexing#sparse-indexes-on-the-columnstore). Set to an empty string to disable sparse indexes; call `ALTER TABLE your_table_name RESET (timescaledb.sparse_index);` to revert to the default selection. |
 | `timescaledb.compress_chunk_time_interval` | INTERVAL    | -                                                                                                                                                                                                                                 | ✖        | EXPERIMENTAL: reduce the total number of {C.CHUNK}s in the {C.COLUMNSTORE} for `table`. If you set `compress_chunk_time_interval`, {C.CHUNK}s added to the {C.COLUMNSTORE} are merged with the previous adjacent {C.CHUNK} within `chunk_time_interval` whenever possible. These {C.CHUNK}s are irreversibly merged. If you call [convert\_to\_rowstore](/reference/timescaledb/hypercore/convert_to_rowstore), merged {C.CHUNK}s are not split up. You can call `compress_chunk_time_interval` independently of other compression settings; `timescaledb.enable_columnstore` is not required.                                                                                                                                                                                                                                                                                                             |
 | `interval`                                 | TEXT    | -                                                                                                                                                                                                                                 | ✖        | Set to a multiple of the [chunk\_time\_interval](/reference/timescaledb/hypertables/set_chunk_time_interval) for `table`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| `timescaledb.direct_compress`              | BOOLEAN | `false`                                                                                                                                                                                                                          | ✖        | Compress data in memory during ingestion and write compressed batches directly to the {C.COLUMNSTORE}, instead of writing uncompressed rows first. Turns on the {C.COLUMNSTORE} if it isn't already enabled. If you set `timescaledb.enable_columnstore = false` in the same statement, `direct_compress` is silently ignored. See [Improve performance with direct compress](/build/data-management/write-data/insert#improve-performance-with-direct-compress).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | `ALTER`                                    | TEXT    |                                                                                                                                                                                                                                   | ✖        | Set a specific column in the columnstore to be `NOT NULL`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | `ADD CONSTRAINT`                           | TEXT    |                                                                                                                                                                                                                                   | ✖        | Add `UNIQUE` constraints to data in the columnstore.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
 

src/content/docs/reference/timescaledb/hypercore/index.mdx

@@ -31,6 +31,9 @@ import HCPolicyWorkflow from '@partials/_hypercore_policy_workflow.mdx';
 - [`add_columnstore_policy()`](/reference/timescaledb/hypercore/add_columnstore_policy): set a policy to automatically move {C.CHUNK}s in a {C.HYPERTABLE}
   to the {C.COLUMNSTORE} when they reach a given age
 - [`remove_columnstore_policy()`](/reference/timescaledb/hypercore/remove_columnstore_policy): remove a {C.COLUMNSTORE} policy from a {C.HYPERTABLE}
+- [`add_compaction_policy()`](/reference/timescaledb/hypercore/add_compaction_policy): set a policy to automatically compact unordered {C.CHUNK}s in
+  the {C.COLUMNSTORE}
+- [`remove_compaction_policy()`](/reference/timescaledb/hypercore/remove_compaction_policy): remove a compaction policy from a {C.HYPERTABLE}
 
 ### Configuration
 

src/content/docs/reference/timescaledb/hypercore/remove_compaction_policy.mdx

@@ -0,0 +1,50 @@
+---
+title: remove_compaction_policy()
+description: Remove a compaction policy from a hypertable
+products: [cloud, self_hosted]
+keywords: [hypercore, columnstore, policies, compaction, remove]
+---
+
+import * as C from "@constants";
+
+import SinceRelease from '@components/SinceRelease.astro';
+
+import ReturnsVoid from '@partials/_returns-void.mdx';
+
+<SinceRelease version="2.29.0" product="tsdb" />
+
+Remove a [compaction policy](/reference/timescaledb/hypercore/add_compaction_policy) from a {C.HYPERTABLE}.
+
+Unordered {C.CHUNK}s are no longer compacted automatically, so queries against them keep the extra sort step needed to
+restore order. To restart automatic compaction, call [add\_compaction\_policy](/reference/timescaledb/hypercore/add_compaction_policy) again. Turning off
+`direct_compress` also removes the policy it created.
+
+## Samples
+
+You see the compaction policies in the [informational views](/reference/timescaledb/informational-views/jobs).
+
+- **Remove the compaction policy from the `metrics` {C.HYPERTABLE}**:
+
+  ```sql
+  SELECT remove_compaction_policy('metrics');
+  ```
+
+## Arguments
+
+The syntax is:
+
+```sql
+SELECT remove_compaction_policy(
+    hypertable = '<hypertable_name>',
+    if_exists = true | false
+);
+```
+
+| Name | Type | Default | Required | Description |
+|--|--|--|--|-|
+|`hypertable`|REGCLASS|-|✔| Name of the {C.HYPERTABLE} to remove the policy from|
+| `if_exists` | BOOLEAN | `false` |✖| Set to `true` so this job fails with a warning rather than an error if a compaction policy does not exist on `hypertable` |
+
+## Returns
+
+<ReturnsVoid />

src/content/docs/reference/timescaledb/hypertables/create_table.mdx

@@ -156,7 +156,8 @@ WITH (
    tsdb.associated_table_prefix = '<prefix>',
    tsdb.orderby = '<column_name> [ASC | DESC] [ NULLS { FIRST | LAST } ] [, ...]',
    tsdb.segmentby = '<column_name> [, ...]',
-   tsdb.sparse_index = '<index>(<column_name>), index(<column_name>)'
+   tsdb.sparse_index = '<index>(<column_name>), index(<column_name>)',
+   tsdb.direct_compress = true | false
 )
 ```
 
@@ -172,6 +173,7 @@ WITH (
 | `tsdb.orderby`                 |TEXT| Descending order on the time column in `table_name`.                                                                                                                                                                                                                                          | ✖| The order in which items are used in the {C.COLUMNSTORE}. Specified in the same way as an `ORDER BY` clause in a `SELECT` query. Setting `tsdb.orderby` automatically creates an implicit sparse index on the `orderby` column: a `firstlast` index since [2.28.0](https://github.com/timescale/timescaledb/releases/tag/2.28.0), `minmax` before.                                                                                                                                                                                                                                                                                                                                                                                       |
 | `tsdb.segmentby`               |TEXT| {C.TIMESCALE_DB} looks at [`pg_stats`](https://www.postgresql.org/docs/current/view-pg-stats.html) and determines an appropriate column based on the data cardinality and distribution. If `pg_stats` is not available, {C.TIMESCALE_DB} looks for an appropriate column from the existing indexes. | ✖| Set the list of columns used to segment data in the {C.COLUMNSTORE} for `table`. An identifier representing the source of the data such as `device_id` or `tags_id` is usually a good candidate.                                                                                                                                                                                                                                                                                                                                                                                                                              |
 |`tsdb.sparse_index`| TEXT | {C.TIMESCALE_DB} evaluates the columns you already have indexed, checks which data types are a good fit for sparse indexing, then creates a sparse index as an optimization.                                                                     | ✖ | Configure the sparse indexes for compressed {C.CHUNK}s. Requires setting `tsdb.orderby`. Supported index types are `bloom(<column>, …)` for equality filters and `minmax(<column>)` for range filters; use a comma-separated list to set multiple. For when each is best, configuration patterns, and restrictions, see [Sparse indexes on the columnstore](/build/performance-optimization/indexing#sparse-indexes-on-the-columnstore). Set to an empty string to disable sparse indexes. |
+| `tsdb.direct_compress`         |BOOLEAN| `false`                                                                                                                                                                                                                                                                                        | ✖                                                           | Compress data in memory during ingestion and write compressed batches directly to the {C.COLUMNSTORE}, instead of writing uncompressed rows first. Only takes effect when the {C.COLUMNSTORE} is enabled. The {C.COLUMNSTORE} is on by default, so no extra setting is required. If you set `tsdb.columnstore = false` in the same statement, `direct_compress` is silently ignored. See [Improve performance with direct compress](/build/data-management/write-data/insert#improve-performance-with-direct-compress). |
 
 ## Returns