fix(ducklake): drain inlined data even when compaction is disabled - #766
Merged
Conversation
PR #765 added ducklake_flush_inlined_data to the CompactionService cycle, but that service is opt-in (disabled by default). So a node that upgrades to the new inlining-off default while leaving compaction off never drains its legacy inline backlog: disabling inlining only stops NEW inlining, it does not flush rows inlined earlier, which stay in the catalog and resident in the DuckLake extension's memory. This is the same end symptom fixed in the sibling ingest service (stuck inline backlog bloating the extension heap). - CompactionService now always starts: full merge/expire/cleanup when enabled, otherwise a lightweight inline-flush-only loop paced by CheckIntervalSec (first run ~1 min after startup so a backlog drains promptly). - Extracted flushInlinedData() shared by the full cycle (step 0) and the flush-only loop; it reapplies S3 settings and is a no-op once nothing is inlined. Bump 11.0.232 -> 11.0.233.
Per-flush Parquet files and DuckLake snapshots accumulate quickly; without periodic merge/expire/cleanup the catalog and file count grow unbounded. Flip compaction.enable default false -> true so maintenance (including the inlined-data flush) runs out of the box. Operators can still opt out explicitly. Bump 11.0.233 -> 11.0.234.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR #765 added
ducklake_flush_inlined_datato theCompactionServicecycle so inlined data is drained to Parquet. ButCompactionServicewas opt-in (compaction.enabledefaulted tofalse).That left a gap: a node that upgrades to the new inlining-off default while keeping compaction disabled never drains its legacy inline backlog. Disabling data inlining only stops new inlining — it does not flush rows inlined earlier. Those rows stay in the catalog and resident in the DuckLake extension's memory, which shows up as steadily growing RSS.
Fix
compaction.enablenow defaults totrue— per-flush Parquet files and snapshots accumulate quickly, so maintenance (merge / expire / cleanup, plus the inlined-data flush) should run out of the box. Operators can still opt out explicitly.CompactionServicenow always starts, even when compaction is disabled:CheckIntervalSec(first run ~1 min after startup so an existing backlog drains promptly).flushInlinedData(), shared by the full cycle (step 0) and the flush-only loop. It reapplies S3 client settings, holds the catalog lock (serialized with writer flushes), and is a cheap no-op once nothing is inlined.Notes
compaction.check_interval_seccadence (default 3600s).Bump
11.0.232->11.0.234.Test plan
compaction.enable=falseagainst a catalog that has aducklake_inlined_data_*backlog; confirm the "Flush inlined data" maintenance log fires and the inline tables drain / RSS drops.