[BUG] ClickHouse system log tables (trace_log/text_log) grow unbounded and can fill disk in default installer configuration

[nazarsavorona]

Description

The n8n installer deploys a ClickHouse container with ClickHouse system logging (especially system.trace_log and system.text_log) effectively unbounded. Over time this causes the ClickHouse data volume to grow rapidly and can fill the host disk (Docker root/overlay2), resulting in service instability/outages. In our case, the ClickHouse data volume reached ~38 GB almost entirely from system.trace_log (~30 GiB) plus other system log tables, even though application data was minimal.

Steps to Reproduce

1. Deploy the stack using the n8n installer defaults (includes ClickHouse container).
2. Let the stack run under normal workload for days/weeks.
3. Observe disk usage on the host increasing steadily (especially /var/lib/docker/volumes/<clickhouse_volume>/_data).
4. Inspect ClickHouse table sizes and see system.trace_log, system.text_log, system.asynchronous_metric_log, and system.metric_log consuming most disk.

Example commands:

• docker system df
• sudo du -sh /var/lib/docker/volumes/* | sort -h | tail -n 30
• docker exec -it <clickhouse_container> clickhouse-client --query "SELECT database, table, formatReadableSize(sum(bytes_on_disk)) size FROM system.parts WHERE active GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 30;"

Expected Behavior

Installer should configure ClickHouse so system logging does not grow unbounded:

• Disable or significantly limit trace_log and text_log by default (or set sensible retention/TTL).
• Keep the host disk from filling due to ClickHouse system log tables.
• Provide safe defaults suitable for long-running production deployments.

Actual Behavior

ClickHouse system log tables grow without bounds, eventually filling Docker storage and the host root filesystem. In our environment:

• Host root partition reached ~97% usage.
• ClickHouse volume localai_langfuse_clickhouse_data grew to ~38 GB.
• Largest tables were in system DB:
  • system.trace_log ~29.97 GiB
  • system.text_log ~2.34 GiB
  • system.asynchronous_metric_log ~1.08 GiB
  • system.metric_log ~803 MiB

Row counts indicated extreme growth:

• trace_log: ~1.94B rows (oldest 2025-12-09, newest 2026-01-27)
• asynchronous_metric_log: ~2.00B rows (oldest 2025-09-16, newest 2026-01-27)
• text_log: ~67M rows (oldest 2025-09-16, newest 2026-01-27)
• metric_log: ~4.2M rows (oldest 2025-12-09, newest 2026-01-27)

Screenshots

N/A (can provide terminal outputs if needed).

Environment

• OS: Ubuntu 24.04
• Using Docker Engine (not Docker Desktop / not WSL)

Additional Context

• Happens consistently over time on a long-running host.
• No unusual debugging flags were intentionally enabled on our side.
• Workaround we used: truncate the system log tables to recover disk space:
  • TRUNCATE TABLE system.trace_log;
  • TRUNCATE TABLE system.text_log;
  • TRUNCATE TABLE system.asynchronous_metric_log;
  • TRUNCATE TABLE system.metric_log; (and related _0/_1 tables)
    This immediately reduced the ClickHouse data volume from ~38G to ~98M and freed ~40G on the host.
• Without a persistent config change, these tables will grow again.

Possible Solution

Set safer ClickHouse defaults in the installer:

1. Disable high-volume system logs by default:

• Provide a config.d override such as:
  • <trace_log remove="remove" />
  • <text_log remove="remove" />

2. If logs are needed, set retention/TTL (e.g., keep only 3–7 days) and/or reduce collection/flush intervals for metric logs.

3. Document the behavior and provide recommended production settings (log retention, volume sizing, monitoring alerts).