Skip to content

fix(ducklake): GC leaked inline tables to stop multi-GB RSS growth - #767

Merged
adubovikov merged 1 commit into
homer11from
fix/ducklake-inline-table-leak
Jun 1, 2026
Merged

fix(ducklake): GC leaked inline tables to stop multi-GB RSS growth#767
adubovikov merged 1 commit into
homer11from
fix/ducklake-inline-table-leak

Conversation

@adubovikov

@adubovikov adubovikov commented Jun 1, 2026

Copy link
Copy Markdown
Member

Root cause

DuckLake leaks ducklake_inlined_data_* physical tables — upstream bug duckdb/ducklake#1065:

  • DuckLake creates a ducklake_inlined_data_<table_id>_<schema_version> table per schema version.
  • schema_version bumps on every DDL — including the SET PARTITIONED BY / SET SORTED BY we issue for each hep_proto_* and otlp_* table on every startup.
  • No maintenance path ever DROPs those tables: flush_inlined_data only DELETEs their rows, expire_snapshots only removes registry rows, cleanup_old_files ignores them.

They accumulate forever, and the DuckLake extension rebuilds an in-memory stats map (DuckLakeCatalog::ConstructStatsMap) over all of them on every catalog refresh → RSS climbs without bound even with a tiny catalog and few Parquet files.

This is the same defect diagnosed and fixed in the sibling ingest service (heap profile pinned ~2.5 GB to ConstructStatsMap fed by ~870 leaked inline tables; dropping the empty ones took RSS from 4.1 GB back to ~300 MB).

Fix

  • Startup GC (GCOrphanInlineTables). Before ATTACH — while the sqlite catalog is still exclusively ours, the same window EnableSQLiteWALMode already uses — drop every empty ducklake_inlined_data_* table and delete its ducklake_inlined_data_tables registry row. Empty ⇒ rows already flushed to Parquet, so the cleanup is lossless. Best-effort, non-fatal. Non-empty inline tables (un-flushed legacy rows) are left untouched.
  • Stop the churn. SET PARTITIONED BY / SET SORTED BY now run only when the table is first created (gated on information_schema.tables), for both HEP (tables.go) and OTLP (otlp_storage.go) tables — not on every boot. Re-issuing them bumped schema_version each restart and spawned a fresh leaked inline table per table per restart.

Bump 11.0.23411.0.235.

Notes

  • sqlite catalog only (the GC SQL is sqlite); homer's DuckLake catalog is always sqlite.
  • Existing nodes drain their backlog at the next restart (GC runs at startup, drops only empty tables). To reclaim immediately: stop the service, back up the catalog .sqlite, DROP the empty ducklake_inlined_data_* tables + their registry rows, VACUUM, restart.

Test plan

  • Start against a catalog with a ducklake_inlined_data_* backlog; confirm the "DuckLake inline GC: dropped N ..." log and that ducklake_inlined_data_tables shrinks to the non-empty set.
  • Restart a few times; confirm the inline-table count stays flat (no churn) and existing tables keep their partitioning/sort order.
  • Fresh catalog: confirm hep_proto_* and otlp_* tables are still created PARTITIONED BY (date) / SORTED BY (timestamp ASC).

DuckLake creates a ducklake_inlined_data_<table_id>_<schema_version> table
per schema version, schema_version bumps on every DDL (incl. our per-table
SET PARTITIONED BY / SET SORTED BY), and no maintenance path ever DROPs them
(flush only DELETEs rows, expire only drops registry rows, cleanup ignores
them). They accumulate forever and the extension rebuilds an in-memory stats
map (DuckLakeCatalog::ConstructStatsMap) over all of them on every catalog
refresh, growing RSS to multiple GB. Upstream bug duckdb/ducklake#1065.

- Startup GC (GCOrphanInlineTables): before ATTACH (exclusive sqlite access,
  same window as EnableSQLiteWALMode) drop every EMPTY ducklake_inlined_data_*
  table and its ducklake_inlined_data_tables registry row. Empty => rows
  already flushed to Parquet, so lossless. Best-effort, non-fatal.
- Stop the churn: SET PARTITIONED BY / SET SORTED BY now run only when the
  HEP/OTLP table is first created (gated on information_schema.tables), not on
  every startup (each re-issue bumped schema_version and spawned another
  leaked inline table per restart).

Bump 11.0.234 -> 11.0.235.
@CLAassistant

CLAassistant commented Jun 1, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@adubovikov
adubovikov merged commit 9c46496 into homer11 Jun 1, 2026
9 of 10 checks passed
@adubovikov
adubovikov deleted the fix/ducklake-inline-table-leak branch June 1, 2026 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants