-
Notifications
You must be signed in to change notification settings - Fork 115
Add per-index compaction controls #475
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
tjgreen42
wants to merge
5
commits into
background-compaction-1-engine
from
background-compaction-2-api
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
56e4681
feat: add per-index compaction controls
tjgreen42 270e7a1
fix: harden per-index compaction controls
tjgreen42 aae56a1
Clarify terminal compaction behavior
tjgreen42 cf03136
fix: precheck compaction index ownership
tjgreen42 39f47d5
refactor: colocate stepped compaction helper
tjgreen42 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # Native compaction controls | ||
|
|
||
| pg_textsearch provides scheduler-independent SQL functions for inspecting and | ||
| compacting one BM25 index: | ||
|
|
||
| - `bm25_level_counts(regclass)` returns the persisted segment count for each | ||
| of the eight LSM levels. | ||
| - `bm25_needs_compaction(regclass)` reports whether any compactable level, | ||
| L0 through L6, has reached `pg_textsearch.segments_per_level`. | ||
| - `bm25_compact(regclass)` runs the full compaction cascade until no | ||
| compactable level remains over threshold. Before its first merge, it | ||
| simulates the complete cascade and rejects any destination-capacity | ||
| failure without changing the index. | ||
| - `bm25_compact_step(regclass)` runs at most one merge batch and returns | ||
| whether a batch ran. | ||
|
|
||
| The two mutating functions require ownership of the index. They open the | ||
| relation with `RowExclusiveLock` and hold the index's `LW_EXCLUSIVE` lock | ||
| while merging. `bm25_compact()` holds that lock for the entire cascade. | ||
| `bm25_compact_step()` releases it after one batch, allowing callers to split | ||
| a cascade across transactions. | ||
|
|
||
| Permanent and unlogged indexes cannot be compacted in a read-only transaction, | ||
| and no index can be compacted during recovery. A local temporary index remains | ||
| available to its owning backend and may be compacted in a read-only | ||
| transaction. | ||
|
|
||
| Partitioned BM25 parent indexes have no physical storage and are rejected by | ||
| all four per-index functions. Call the functions on the physical indexes of | ||
| individual partitions instead. | ||
|
|
||
| L7 is the terminal level and is not itself compactable. | ||
| `bm25_needs_compaction()` therefore considers only L0 through L6. If a full L7 | ||
| prevents promotion from L6, `bm25_compact()` rejects the complete cascade | ||
| before any physical merge, even when valid lower-level debt precedes the | ||
| blocked L6 batch. `bm25_compact_step()` plans only its next batch, so it may | ||
| merge valid lower-level debt before a later call encounters and rejects the | ||
| blocked L6 batch. Operators must resolve such a terminal layout rather than | ||
| retrying it as ordinary compaction debt. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,202 @@ | ||
| /* | ||
| * Copyright (c) 2025-2026 Tiger Data, Inc. | ||
| * Licensed under the PostgreSQL License. See LICENSE for details. | ||
| * | ||
| * compaction.c - BM25 index compaction inspection and control | ||
| */ | ||
| #include <postgres.h> | ||
|
|
||
| #include <access/relation.h> | ||
| #include <access/xlog.h> | ||
| #include <catalog/objectaccess.h> | ||
| #include <catalog/pg_class.h> | ||
| #include <catalog/pg_type.h> | ||
| #include <miscadmin.h> | ||
| #include <utils/acl.h> | ||
| #include <utils/array.h> | ||
| #include <utils/lsyscache.h> | ||
|
|
||
| #include "access/am.h" | ||
| #include "constants.h" | ||
| #include "index/metapage.h" | ||
| #include "index/state.h" | ||
| #include "segment/merge.h" | ||
|
|
||
| /* | ||
| * Open a bm25 index by OID, validating that it is in fact a bm25 | ||
| * index and (optionally) that the caller owns it. | ||
| */ | ||
| static Relation | ||
| tp_open_bm25_index(Oid indexoid, LOCKMODE lockmode, bool need_owner) | ||
| { | ||
| Relation index_rel; | ||
|
|
||
| /* | ||
| * Reject nonowners before queuing for a heavyweight lock. Recheck after | ||
| * opening because ALTER OWNER can complete while this caller waits. | ||
| */ | ||
| if (need_owner && | ||
| !object_ownercheck(RelationRelationId, indexoid, GetUserId())) | ||
| { | ||
| char *relname = get_rel_name(indexoid); | ||
|
|
||
| if (relname == NULL) | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_UNDEFINED_OBJECT), | ||
| errmsg("relation with OID %u does not exist", indexoid))); | ||
| aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_INDEX, relname); | ||
| } | ||
|
|
||
| index_rel = relation_open(indexoid, lockmode); | ||
|
|
||
| if (index_rel->rd_indam == NULL || | ||
| index_rel->rd_indam->ambuild != tp_build) | ||
| { | ||
| char *relname = pstrdup(RelationGetRelationName(index_rel)); | ||
|
|
||
| relation_close(index_rel, lockmode); | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_WRONG_OBJECT_TYPE), | ||
| errmsg("\"%s\" is not a bm25 index", relname))); | ||
| } | ||
|
|
||
| if (index_rel->rd_rel->relkind == RELKIND_PARTITIONED_INDEX) | ||
| { | ||
| char *relname = pstrdup(RelationGetRelationName(index_rel)); | ||
|
|
||
| relation_close(index_rel, lockmode); | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_WRONG_OBJECT_TYPE), | ||
| errmsg("\"%s\" is a partitioned bm25 index", relname), | ||
| errhint("Use a physical partition index instead."))); | ||
| } | ||
|
|
||
| if (need_owner && | ||
| !object_ownercheck(RelationRelationId, indexoid, GetUserId())) | ||
| { | ||
| char *relname = pstrdup(RelationGetRelationName(index_rel)); | ||
|
|
||
| relation_close(index_rel, lockmode); | ||
| aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_INDEX, relname); | ||
| } | ||
|
|
||
| return index_rel; | ||
| } | ||
|
|
||
| PG_FUNCTION_INFO_V1(tp_level_counts); | ||
|
|
||
| Datum | ||
| tp_level_counts(PG_FUNCTION_ARGS) | ||
| { | ||
| Oid indexoid = PG_GETARG_OID(0); | ||
| Relation index_rel; | ||
| TpIndexMetaPageData *metap; | ||
| Datum elems[TP_MAX_LEVELS]; | ||
| ArrayType *result; | ||
| int i; | ||
|
|
||
| index_rel = tp_open_bm25_index(indexoid, AccessShareLock, false); | ||
|
|
||
| metap = tp_get_metapage(index_rel); | ||
| for (i = 0; i < TP_MAX_LEVELS; i++) | ||
| elems[i] = Int32GetDatum((int32)metap->level_counts[i]); | ||
| pfree(metap); | ||
|
|
||
| relation_close(index_rel, AccessShareLock); | ||
|
|
||
| result = construct_array( | ||
| elems, TP_MAX_LEVELS, INT4OID, sizeof(int32), true, TYPALIGN_INT); | ||
| PG_RETURN_ARRAYTYPE_P(result); | ||
| } | ||
|
|
||
| PG_FUNCTION_INFO_V1(tp_compact_index); | ||
|
|
||
| Datum | ||
| tp_compact_index(PG_FUNCTION_ARGS) | ||
| { | ||
| Oid indexoid = PG_GETARG_OID(0); | ||
| Relation index_rel; | ||
| TpLocalIndexState *index_state; | ||
|
|
||
| if (RecoveryInProgress()) | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION), | ||
| errmsg("cannot compact a bm25 index during recovery"))); | ||
|
|
||
| index_rel = tp_open_bm25_index(indexoid, RowExclusiveLock, true); | ||
|
|
||
| if (!RelationUsesLocalBuffers(index_rel)) | ||
| PreventCommandIfReadOnly("bm25 index compaction"); | ||
|
|
||
| index_state = tp_get_local_index_state(indexoid); | ||
| if (index_state == NULL) | ||
| { | ||
| char *relname = pstrdup(RelationGetRelationName(index_rel)); | ||
|
|
||
| relation_close(index_rel, RowExclusiveLock); | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_INTERNAL_ERROR), | ||
| errmsg("could not get index state for \"%s\"", relname))); | ||
| } | ||
|
|
||
| tp_acquire_index_lock(index_state, LW_EXCLUSIVE); | ||
| PG_TRY(); | ||
| { | ||
| tp_compaction_preflight(index_rel); | ||
| tp_maybe_compact_level(index_rel, 0); | ||
| } | ||
| PG_FINALLY(); | ||
| { | ||
| tp_release_index_lock(index_state); | ||
| relation_close(index_rel, RowExclusiveLock); | ||
| } | ||
| PG_END_TRY(); | ||
|
|
||
| PG_RETURN_VOID(); | ||
| } | ||
|
|
||
| PG_FUNCTION_INFO_V1(tp_compact_index_step); | ||
|
|
||
| Datum | ||
| tp_compact_index_step(PG_FUNCTION_ARGS) | ||
| { | ||
| Oid indexoid = PG_GETARG_OID(0); | ||
| Relation index_rel; | ||
| TpLocalIndexState *index_state; | ||
| bool batch_ran; | ||
|
|
||
| if (RecoveryInProgress()) | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_READ_ONLY_SQL_TRANSACTION), | ||
| errmsg("cannot compact a bm25 index during recovery"))); | ||
|
|
||
| index_rel = tp_open_bm25_index(indexoid, RowExclusiveLock, true); | ||
|
|
||
| if (!RelationUsesLocalBuffers(index_rel)) | ||
| PreventCommandIfReadOnly("bm25 index compaction"); | ||
|
|
||
| index_state = tp_get_local_index_state(indexoid); | ||
| if (index_state == NULL) | ||
| { | ||
| char *relname = pstrdup(RelationGetRelationName(index_rel)); | ||
|
|
||
| relation_close(index_rel, RowExclusiveLock); | ||
| ereport(ERROR, | ||
| (errcode(ERRCODE_INTERNAL_ERROR), | ||
| errmsg("could not get index state for \"%s\"", relname))); | ||
| } | ||
|
|
||
| tp_acquire_index_lock(index_state, LW_EXCLUSIVE); | ||
| PG_TRY(); | ||
| { | ||
| batch_ran = tp_compact_step(index_rel); | ||
| } | ||
| PG_FINALLY(); | ||
| { | ||
| tp_release_index_lock(index_state); | ||
| relation_close(index_rel, RowExclusiveLock); | ||
| } | ||
| PG_END_TRY(); | ||
|
|
||
| PG_RETURN_BOOL(batch_ran); | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.