[server] support database level buckets limit #2126

caozhen1937 · 2025-12-08T13:04:54Z

Purpose

Linked issue: close #1993

Brief change log

Tests

API and Format

Documentation

swuferhong

Thanks for your contributions. Overall, LGTM. However, there's one scenario we need to restrict—this is actually the reason this issue was raised. Specifically, when auto-partitioning is enabled, we should enforce a limit based on retain partitions * bucket num per partition; otherwise, the total buckets could end up with an excessively large number of retained partitions.

swuferhong · 2025-12-09T02:16:43Z

fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java

+                        maxBucketNum,
+                        dynamicConfigManager.describeConfigs(),
+                        tablePath.getDatabaseName());
+        validateTableDescriptor(tableToCreate, maxBucketNum, dbLevelMaxBucket);


We only need to pass the value returned by DatabaseLimitResolver.resolveMaxBucketForDb into validateTableDescriptor, so the separate maxBucketNum parameter can be removed. Additionally, could we rename the return value of DatabaseLimitResolver.resolveMaxBucketForDb to maxBucketNum for clarity?

@swuferhong maxBucketNum is still used to represent the maximum limit of the cluster. I will use two variables to distinguish them: maxBucketNumOfDb and maxBucketNumOfCluster. How do you think?

@swuferhong maxBucketNum is still used to represent the maximum limit of the cluster. I will use two variables to distinguish them: maxBucketNumOfDb and maxBucketNumOfCluster. How do you think?

What concerns me most is the implementation. Theoretically, the cluster-level logic should be holistic—when creating a table, it should first check how many buckets currently exist in the cluster and then determine whether the table can be created. However, the current cluster-level implementation actually checks individual tables during creation.

I think it's reasonable to separate these two aspects, and future expansion maybe consider for the cluster level limit.

fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java

fluss-server/src/main/java/org/apache/fluss/server/utils/DatabaseLimitResolver.java

fluss-server/src/main/java/org/apache/fluss/server/utils/TableDescriptorValidation.java

swuferhong · 2025-12-14T11:55:48Z

Hi, @caozhen1937 LGTM now. Could you add a test in Flink side call as procedure?

wuchong

Thanks @caozhen1937 for the contribution! The PR addresses an important use case, but I have a few concerns that we should address :

Database-level bucket limit logic is incomplete
Currently, the check only validates whether a single table exceeds a per-table bucket limit. However, the intended requirement is to enforce a total bucket quota across all tables within a database.
To support this, we should maintain a database → totalBucketCount mapping in CoordinatorContext. This state must be:
- Updated on table/partition creation and deletion
- Don't need to persist the state as it can be restored from table bucket information
Backward compatibility must be preserved
The PR removes the existing max.bucket.num configuration, which breaks compatibility. This should be reverted. I found this configuration name is quite confused with the configs introduced in this PR, so we can rename the old config to clarify:
- max.bucket.num → bucket.table.default-limit
- max.partition.num → partition.table.default-limit
Configuration key design
Using ad-hoc, non-deterministic keys like database.limit.xxx.max.bucket.num makes documentation and validation difficult.
A cleaner approach is to use:
- A single map-style config: bucket.database.limits = db1:1000,db2:3000
- A fallback default: bucket.database.default-limit = 500
  We can use alter config with APPEND/SUBTRACT alter type to update or remove per-database limit.
Missing prerequisites for release
Two additional issues should be created and completed before this feature is released:
- Implement a CALL procedure in Flink SQL to dynamically alter cluster-level configs (e.g., database bucket limits)
- Add comprehensive documentation covering:
  - Configuration semantics
  - Default vs. per-database limits
  - How quota enforcement works

What do you think about this @caozhen1937 @swuferhong ?

Thanks again for the great work. Looking forward to the updates!

wuchong · 2025-12-21T07:05:44Z

fluss-client/src/test/java/org/apache/fluss/client/admin/FlussAdminITCase.java

+
+        assertThatThrownBy(() -> admin.createTable(tablePath, tooManyBuckets, false).get())
+                .cause()
+                .isInstanceOf(TooManyBucketsException.class);


assert the message as well, the message should be clear to user the given database has exceed the allowed number of buckets.

wuchong · 2025-12-21T07:06:13Z

fluss-client/src/test/java/org/apache/fluss/client/admin/FlussAdminITCase.java

+                                                tablePath, newPartitionSpec("age", "3"), false)
+                                        .get())
+                .cause()
+                .isInstanceOf(TooManyBucketsException.class);


wuchong · 2025-12-21T07:20:54Z

fluss-server/src/main/java/org/apache/fluss/server/coordinator/MetadataManager.java

+            int maxBucketNumOfDb =
+                    DatabaseLimitResolver.resolveMaxBucketForDb(
+                            maxBucketNumOfCluster,
+                            dynamicConfigManager.describeConfigs(),
+                            tablePath.getDatabaseName());
+            if (totalBuckets > maxBucketNumOfDb) {
                throw new TooManyBucketsException(
                        String.format(
-                                "Adding partition '%s' would result in %d total buckets for table %s, exceeding the maximum of %d buckets.",
+                                "Adding partition '%s' would result in %d total buckets for table %s, exceeding the database-level maximum of %d buckets.",
                                partition.getPartitionName(),
                                totalBuckets,
                                tablePath,
-                                maxBucketNum));
+                                maxBucketNumOfDb));


It seems we simply drops the support of maxBucketNumOfCluster or maxBucketNum. This is backward imcompatible, we shouldn't remove them.

swuferhong · 2025-12-21T08:31:45Z

Hi, @wuchong. A small input regarding "Implement a CALL procedure in Flink SQL to dynamically alter cluster-level configs (e.g., database bucket limits)": the PR #2178 will support using a CALL procedure to dynamically set cluster-level configurations. However, I'm still testing the parameters introduced in #2178, and I'll leave a comment on the PR once testing is complete.

caozhen1937 force-pushed the FLUSS_ISSUE_1993 branch 2 times, most recently from 96a8077 to 0894e8a Compare December 8, 2025 15:15

[server] support database level buckets limit

2e48f97

caozhen1937 force-pushed the FLUSS_ISSUE_1993 branch from 0894e8a to 2e48f97 Compare December 8, 2025 15:18

swuferhong reviewed Dec 9, 2025

View reviewed changes

fix yunhong's comments

bfe6ab0

caozhen1937 force-pushed the FLUSS_ISSUE_1993 branch from fd0fd75 to bfe6ab0 Compare December 13, 2025 05:23

caozhen1937 requested a review from swuferhong December 13, 2025 06:36

add flink it case

db1c130

wuchong requested changes Dec 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[server] support database level buckets limit #2126

[server] support database level buckets limit #2126

Uh oh!

caozhen1937 commented Dec 8, 2025

Uh oh!

swuferhong left a comment

Uh oh!

swuferhong Dec 9, 2025

Uh oh!

caozhen1937 Dec 13, 2025

Uh oh!

swuferhong Dec 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swuferhong commented Dec 14, 2025

Uh oh!

wuchong left a comment

Uh oh!

wuchong Dec 21, 2025

Uh oh!

wuchong Dec 21, 2025

Uh oh!

wuchong Dec 21, 2025

Uh oh!

swuferhong commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[server] support database level buckets limit #2126

Are you sure you want to change the base?

[server] support database level buckets limit #2126

Uh oh!

Conversation

caozhen1937 commented Dec 8, 2025

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

swuferhong left a comment

Choose a reason for hiding this comment

Uh oh!

swuferhong Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

caozhen1937 Dec 13, 2025

Choose a reason for hiding this comment

Uh oh!

swuferhong Dec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

swuferhong commented Dec 14, 2025

Uh oh!

wuchong left a comment

Choose a reason for hiding this comment

Uh oh!

wuchong Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

wuchong Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

wuchong Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

swuferhong commented Dec 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants