fix(web-analytics): use lightweight replica sync before partition swap

[andyzzhao]

Problem

The web analytics pre-aggregation assets follow the pattern: recreate staging table, bulk insert, SYSTEM SYNC REPLICA on every data node, then ALTER TABLE ... REPLACE PARTITION FROM staging. The sync exists so that whichever host later executes the swap has all staging parts locally.

A plain SYSTEM SYNC REPLICA waits for the replica's entire replication queue, which right after a bulk insert includes the freshly scheduled background merges. The job blocks on .result() for the slowest replica, so every run stalls for tens of seconds (sometimes minutes) of pure waiting in system.query_log, with zero bytes read and zero CPU. This pattern showed up as one of the largest "long wall time, no work" query groups on the cluster, hundreds of executions per day across the hourly assets.

The merges are irrelevant to the swap: REPLACE PARTITION FROM clones the staging part set as it exists, merged or not, and readers of AggregatingMergeTree tables must aggregate across parts anyway.

Changes

sync_partitions_on_replicas now issues SYSTEM SYNC REPLICA <table> LIGHTWEIGHT. Lightweight mode waits only for data-movement queue entries (GET_PART, ATTACH_PART, DROP_RANGE, REPLACE_RANGE, DROP_PART), i.e. exactly "all inserted parts are present on this replica", and skips waiting on merges and mutations. ClickHouse itself uses lightweight mode internally for the same parts-must-be-present barrier, and its source notes that plain sync differs precisely in also waiting for merges.

The STRICT syncs in the deletes DAG and overrides manager are deliberate (one asserts an empty queue afterwards) and are not touched.

How did you test this code?

I am an agent; automated tests only:

• New test_sync_partitions_on_replicas_uses_lightweight_sync in products/web_analytics/dags/tests/test_web_preaggregated_utils.py, following the file's mock-cluster pattern (asserts the exact SQL executed per host and that the job waits on the futures).
• Full test_web_preaggregated_utils.py passes (7 tests).

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

• Claude Code session: found this while sweeping query_log_archive for queries with long wall time but near-zero CPU and no cluster-load signature; the staging-table sync was the single most frequent such pattern. Verified in the ClickHouse 26.3 source that lightweight sync waits exactly for part fetches and that the subsequent REPLACE PARTITION does not depend on merges.
• Considered and rejected: leaving full sync but adding a timeout (still wastes the wait and adds a failure mode), and dropping the sync entirely (the swap-executing host genuinely needs all staging parts present, and the executing host is chosen arbitrarily).

[greptile-apps]

_{Reviews (1): Last reviewed commit: "fix(web-analytics): use lightweight repl..." | Re-trigger Greptile}