fix(ducklake): disable data inlining by default, flush it in compaction, batch Line Protocol writes - #765
Merged
Conversation
…ion + batch LP writes
Ports the memory/catalog-bloat fixes proven in the sibling ingest
service (hepic-lake-ingest) to Homer's DuckLake stack. Same storage
class (DuckDB/DuckLake + sqlite catalog + Parquet); the HEP write path
here is already well-batched (Appender + double-buffer + bulk flush),
but three gaps remained:
1. Data inlining default. DuckLakeConfig.DataInliningRowLimit defaulted
to -1 ("leave DuckLake's own default", which inlines ~10-row writes
into the catalog DB). Under streaming Line Protocol / OTLP / low-
volume HEP subtypes this turns the catalog into the dominant memory +
disk consumer (an 800 MB sqlite catalog backing only a few dozen
Parquet files, multi-GB RSS when DuckLake mirrors it in memory).
Default is now 0 (inlining off, always write Parquet). -1 and >0 are
still honoured for operators who want them.
2. No inline flush in maintenance. The CompactionService ran merge /
expire / cleanup / delete-orphaned but never ducklake_flush_inlined_data,
so anything already inlined (or inlined by an operator who re-enables
it) stayed in the catalog forever. Added a flush step at the start of
the maintenance cycle (before merge, so merge/expire act on the
freshly written Parquet). No-op when inlining is disabled.
3. Line Protocol micro-commits. The generic LP path issued one prepared
stmt.ExecContext per row = one DuckLake transaction (snapshot + tiny
write) per row. Replaced with chunked multi-row INSERT ... VALUES
(500 rows/statement), collapsing the per-row transaction/snapshot
churn by up to 500x. hep_proto_* LP and OTLP already batch per
request and are unchanged.
version.go is intentionally untouched — Homer's version is tag-driven
(version-sync.yml updates it from the release tag).
|
|
2 tasks
n0obHere
pushed a commit
to n0obHere/homer
that referenced
this pull request
Jun 22, 2026
PR sipcapture#765 added ducklake_flush_inlined_data to the CompactionService cycle, but that service is opt-in (disabled by default). So a node that upgrades to the new inlining-off default while leaving compaction off never drains its legacy inline backlog: disabling inlining only stops NEW inlining, it does not flush rows inlined earlier, which stay in the catalog and resident in the DuckLake extension's memory. This is the same end symptom fixed in the sibling ingest service (stuck inline backlog bloating the extension heap). - CompactionService now always starts: full merge/expire/cleanup when enabled, otherwise a lightweight inline-flush-only loop paced by CheckIntervalSec (first run ~1 min after startup so a backlog drains promptly). - Extracted flushInlinedData() shared by the full cycle (step 0) and the flush-only loop; it reapplies S3 settings and is a no-op once nothing is inlined. Bump 11.0.232 -> 11.0.233.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a class of DuckLake memory / catalog-bloat issues in Homer's ingest stack (DuckDB/DuckLake + sqlite catalog + Parquet). The HEP write path is already well-batched (Appender + double-buffer + bulk flush) and is left untouched — but three real gaps remained:
1. Data inlining was effectively on by default
DuckLakeConfig.DataInliningRowLimitdefaulted to-1= "leave DuckLake's own default", which inlines small (~10-row) writes into the sqlite catalog instead of Parquet. Under streaming Line Protocol / OTLP / low-volume HEP subtypes this makes the catalog the dominant memory + disk consumer — the classic symptom is an 800 MB sqlite catalog backing only a few dozen Parquet files, with multi-GB RSS once DuckLake mirrors it in memory.0(inlining off, always write Parquet).-1(DuckLake default) and>0(custom threshold) are still honoured for operators who explicitly want them.2. CompactionService never flushed inlined data
The maintenance cycle ran
merge->expire->cleanup->delete_orphanedbut neverducklake_flush_inlined_data, so anything already inlined (or inlined by an operator who re-enables it) stayed in the catalog forever.3. Line Protocol generic path did per-row micro-commits
The generic LP ingest path issued one
stmt.ExecContextper row = one DuckLake transaction (snapshot + tiny write) per row — a micro-commit storm under load.INSERT ... VALUES(500 rows/statement), collapsing per-row transaction/snapshot churn by up to 500x.hep_proto_*LP and OTLP already batch per request and are unchanged.Not changed
src/version.go— Homer's version is tag-driven (version-sync.ymlrewrites it from the release tag), so the bump happens at release time, not in this PR.Test plan
go build ./config/... ./lineprotoreceiver/... ./writer/...go vet ./config/... ./lineprotoreceiver/...ducklake_snapshots) per ingested batch.ducklake_inlined_data_*growth) over a sustained ingest window.Flush inlined dataeach maintenance cycle.