Skip to content

fix(ducklake): drain inlined data even when compaction is disabled - #766

Merged
adubovikov merged 2 commits into
homer11from
fix/drain-inlined-data-without-compaction
May 31, 2026
Merged

fix(ducklake): drain inlined data even when compaction is disabled#766
adubovikov merged 2 commits into
homer11from
fix/drain-inlined-data-without-compaction

Conversation

@adubovikov

@adubovikov adubovikov commented May 31, 2026

Copy link
Copy Markdown
Member

Problem

PR #765 added ducklake_flush_inlined_data to the CompactionService cycle so inlined data is drained to Parquet. But CompactionService was opt-in (compaction.enable defaulted to false).

That left a gap: a node that upgrades to the new inlining-off default while keeping compaction disabled never drains its legacy inline backlog. Disabling data inlining only stops new inlining — it does not flush rows inlined earlier. Those rows stay in the catalog and resident in the DuckLake extension's memory, which shows up as steadily growing RSS.

Fix

  • compaction.enable now defaults to true — per-flush Parquet files and snapshots accumulate quickly, so maintenance (merge / expire / cleanup, plus the inlined-data flush) should run out of the box. Operators can still opt out explicitly.
  • CompactionService now always starts, even when compaction is disabled:
    • compaction enabled -> full merge / expire / cleanup cycle (unchanged);
    • compaction disabled -> a lightweight inline-flush-only loop paced by CheckIntervalSec (first run ~1 min after startup so an existing backlog drains promptly).
  • Extracted flushInlinedData(), shared by the full cycle (step 0) and the flush-only loop. It reapplies S3 client settings, holds the catalog lock (serialized with writer flushes), and is a cheap no-op once nothing is inlined.

Notes

  • No new config knobs; the flush-only loop reuses the existing compaction.check_interval_sec cadence (default 3600s).

Bump 11.0.232 -> 11.0.234.

Test plan

  • Fresh start with defaults: confirm the full compaction cycle runs (compaction now on by default).
  • Run with explicit compaction.enable=false against a catalog that has a ducklake_inlined_data_* backlog; confirm the "Flush inlined data" maintenance log fires and the inline tables drain / RSS drops.

PR #765 added ducklake_flush_inlined_data to the CompactionService cycle,
but that service is opt-in (disabled by default). So a node that upgrades
to the new inlining-off default while leaving compaction off never drains
its legacy inline backlog: disabling inlining only stops NEW inlining, it
does not flush rows inlined earlier, which stay in the catalog and
resident in the DuckLake extension's memory. This is the same end symptom
fixed in the sibling ingest service (stuck inline backlog bloating the
extension heap).

- CompactionService now always starts: full merge/expire/cleanup when
  enabled, otherwise a lightweight inline-flush-only loop paced by
  CheckIntervalSec (first run ~1 min after startup so a backlog drains
  promptly).
- Extracted flushInlinedData() shared by the full cycle (step 0) and the
  flush-only loop; it reapplies S3 settings and is a no-op once nothing is
  inlined.

Bump 11.0.232 -> 11.0.233.
@CLAassistant

CLAassistant commented May 31, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Per-flush Parquet files and DuckLake snapshots accumulate quickly; without
periodic merge/expire/cleanup the catalog and file count grow unbounded.
Flip compaction.enable default false -> true so maintenance (including the
inlined-data flush) runs out of the box. Operators can still opt out
explicitly.

Bump 11.0.233 -> 11.0.234.
@adubovikov
adubovikov merged commit 2efd323 into homer11 May 31, 2026
9 of 10 checks passed
@adubovikov
adubovikov deleted the fix/drain-inlined-data-without-compaction branch May 31, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants