Skip to content

Support concurrent partition detach - #8802

Open
Andrey Borodin (x4m) wants to merge 6 commits into
citusdata:mainfrom
x4m:detach-partition-concurrently
Open

Support concurrent partition detach#8802
Andrey Borodin (x4m) wants to merge 6 commits into
citusdata:mainfrom
x4m:detach-partition-concurrently

Conversation

@x4m

Copy link
Copy Markdown

Support ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY for distributed
partitioned tables.

PostgreSQL implements concurrent detach using two transactions, so shard
commands cannot run inside worker_apply_inter_shard_ddl_command(). This
change instead builds top-level nontransactional commands for each colocated
pair of parent and partition shards. The detached table remains available as
an independent distributed table and can be dropped separately.

DETACH PARTITION ... FINALIZE propagates a full concurrent detach to the
workers. This lets it finish an operation canceled after PostgreSQL committed
the coordinator's pending-detach state but before Citus started worker DDL.

The tests cover a normal detach and that interruption window, including the
catalog state on both workers before and after FINALIZE.

One design question remains: nontransactional DDL can fail on only some
placements after the coordinator has completed. The patch currently uses the
same partial-failure warning mechanism as other concurrent Citus DDL. Should
we also make FINALIZE an idempotent placement-reconciliation operation before
merging this?

Fixes #5264

Dropping a large distributed partition can spend substantial time
unlinking shard storage.  PostgreSQL's two-phase concurrent detach lets
users first remove the partition from query routing and drop the resulting
standalone table separately.

Run shard detach commands as top-level nontransactional DDL, since
PostgreSQL implements DETACH PARTITION CONCURRENTLY using internal commits
and they cannot go through worker_apply_inter_shard_ddl_command.  Propagate
FINALIZE as a concurrent shard detach so an interrupted coordinator
operation can be completed.

Add functional coverage and an isolation test that cancels the operation
after its first phase, verifies the intermediate coordinator and worker
state, and completes it with FINALIZE.

Discussion: citusdata#5264

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for PostgreSQL 14’s ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY (and recovery via ... FINALIZE) for distributed partitioned tables in Citus. This fits into Citus’ DDL propagation layer by introducing a non-transactional worker execution path for this specific DDL, since PostgreSQL implements concurrent detach using multiple transactions.

Changes:

  • Implement distributed concurrent partition detach by generating top-level, non-transactional per-(parent shard, partition shard) worker commands instead of using worker_apply_inter_shard_ddl_command().
  • Treat DETACH PARTITION ... FINALIZE as a mechanism to run the full concurrent detach on workers (to complete operations interrupted after coordinator pending-detach state was committed).
  • Add regression + isolation coverage for successful detach and the interruption/FINALIZE recovery window, and wire tests into schedules.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/backend/distributed/commands/table.c Builds per-shard top-level DETACH PARTITION ... CONCURRENTLY tasks and routes FINALIZE through the same non-transactional worker path.
src/test/regress/sql/pg14.sql Updates PG14 compatibility test to assert concurrent detach works on distributed partitioned tables.
src/test/regress/expected/pg14.out Expected output updates for the new concurrent-detach behavior in pg14.sql.
src/test/regress/sql/detach_partition_concurrently.sql New multi-node regression test validating detach behavior and post-detach usability of the detached table.
src/test/regress/expected/detach_partition_concurrently.out Expected output for the new regression test.
src/test/regress/spec/isolation_detach_partition_concurrently.spec New isolation test covering cancellation after coordinator pending-detach commit and recovery via FINALIZE.
src/test/regress/expected/isolation_detach_partition_concurrently.out Expected isolation output for the new spec.
src/test/regress/multi_schedule Schedules the new detach_partition_concurrently regression test.
src/test/regress/isolation_schedule Schedules the new isolation test for concurrent detach recovery.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comment thread src/backend/distributed/commands/table.c Outdated
@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.74699% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.72%. Comparing base (c5284e1) to head (5c69a92).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8802      +/-   ##
==========================================
- Coverage   88.74%   88.72%   -0.02%     
==========================================
  Files         289      289              
  Lines       64992    65069      +77     
  Branches     8200     8206       +6     
==========================================
+ Hits        57676    57735      +59     
- Misses       4950     4963      +13     
- Partials     2366     2371       +5     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A nontransactional detach can leave placements attached, pending, or
already detached after a partial failure. Running the same command on
every placement cannot recover all three states, and changing the
coordinator first can make a retry impossible.

Reconcile each placement before changing the coordinator: use
CONCURRENTLY for attached shards, FINALIZE for pending shards, and skip
detached shards. Add coverage for workers that completed different
amounts of work and for retrying after cancellation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Postgres 14 support - Missing features on Citus

2 participants