Add broker config and update compatibility

Author: nodece

Diff for: pip/pip-398.md

@@ -1,4 +1,4 @@
-# PIP-398: Subscription replication on the namespace and topic levels
+# PIP-398: Subscription replication on the broker, namespace and topic levels
 
 # Background knowledge
 
@@ -18,36 +18,53 @@ large-scale deployments.
 The key motivation behind this PIP is to simplify subscription replication configuration, especially in failover
 scenarios. When a main cluster goes down and a backup cluster is activated, ensuring that subscription states are
 consistently replicated across clusters is critical for failover scenarios. By extending the replication configuration
-to the namespace and topic levels, the system reduces the need for explicit consumer-level configuration.
+to the broker, namespace and topic levels, the system reduces the need for explicit consumer-level configuration.
 
 # Goals
 
 ## In Scope
 
-The PIP aims to provide management of subscription replication at the namespace and topic levels using the Pulsar Admin
-CLI and API.
+The PIP aims to provide management of subscription replication at the broker, namespace and topic levels using the
+Pulsar Admin CLI and API.
 
 # Out scope
 
 This feature depends on https://github.com/apache/pulsar/pull/23757, which makes the subscription replication nullable
-on the consumer level. Otherwise, the namespace and topic levels will be ignored.
+on the consumer level. Otherwise, the broker, namespace and topic levels will be ignored.
 
 # High Level Design
 
-Introduces a configuration used for enabling subscription replication on the namespace and topic levels, when enabled,
-all consumers under the topic will automatically replicate their subscription states to remote clusters.
+Introduces a configuration used for enabling subscription replication on the broker, namespace and topic levels, when
+enabled, all consumers under the topic will automatically replicate their subscription states to remote clusters.
 
 The priority for the subscription replication configuration is as follows:
 
-- consumer level > topic level > namespace level.
-- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic and namespace levels are
+- consumer level > topic level > namespace level > broker level.
+- If `replicateSubscriptionState` is set at the consumer level, configurations at the topic, namespace, and broker levels are
   ignored.
 - If set at the topic level, the namespace-level configuration is ignored.
+- If set at the namespace level, the broker-level configuration is ignored.
 
 # Detailed Design
 
 ## Design & Implementation Details
 
+### Broker level
+
+Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.broker.ServiceConfiguration` class
+to control subscription replication at the broker level:
+```java
+public class ServiceConfiguration implements PulsarConfiguration {
+    @FieldContext(
+            category = CATEGORY_SERVER,
+            required = false,
+            dynamic = true,
+            doc = "The default value for replicating subscription state."
+    )
+    private Boolean replicate_subscriptions_state;
+}
+```
+
 ### Namespace level
 
 1. Add the field `Boolean replicate_subscriptions_state` to the `org.apache.pulsar.common.policies.data.Policies` class
@@ -91,19 +108,20 @@ The priority for the subscription replication configuration is as follows:
 
 ### Consumer level
 
-We currently support the `replicateSubscriptionState` flag at the consumer level, which can be set to `true`, `false`.
-If `true`, we persist `pulsar.replicated.subscription=1L` to the cursor properties, and if `false`, we remove that.
+We currently support the `replicateSubscriptionState` flag for new subscription at the consumer level, which can be set
+to `true` or `false`. If `true`, we persist `pulsar.replicated.subscription=1L` to the cursor properties, and if
+`false`, we remove that.
 
 To keep the consumer-level configuration, we need to consider the `false` case, so propose using
 `pulsar.replicated.subscription=0L` instead of removing the property.
 
-The consumer-level configuration takes precedence over both the topic and namespace levels, ensuring that
+The consumer-level configuration takes precedence over both the topic, broker and namespace levels, ensuring that
 consumer-specific configurations can still override the more general settings.
 
 ### Subscription replication applied
 
-When the subscription replication is changed on the namespace or topic level, the subscription replication will be
-applied to all consumers under the topic.
+When the subscription replication is changed on the broker, namespace or topic level, the subscription replication will
+be applied to all consumers under the topic.
 
 Please notice that the `org.apache.pulsar.client.admin.Topics.setReplicatedSubscriptionStatus()` can override the
 subscription replication configuration for the special subscription, and equals the consumer level.
@@ -114,7 +132,7 @@ subscription replication configuration for the special subscription, and equals
 
 ### Client API
 
-Change the set replicated subscription status method in the `org.apache.pulsar.client.admin.Topics` interface to accept the Boolean type:
+Change the set replicated subscription status method in the `org.apache.pulsar.client.admin.Topics` interface to accept the `Boolean` type:
 
 ```java
 void setReplicatedSubscriptionStatus(String topic, String subName, Boolean enabled) throws PulsarAdminException;
@@ -212,21 +230,38 @@ Both write and read operations require the necessary permissions, which already
 
 ## Upgrade
 
-The existing subscription with `replicateSubscriptionSate=false`,the broker will use the policy from the topic or namespace level.
-
-New subscriptions will be followed as usual.
+Because the consumer's `replicateSubscriptionSate=false` does not persist to the cursor properties, it leads to
+compatibility issues:
+
+- When the existing subscription with `replicateSubscriptionSate=false` on the consumer level, which uses the policy
+  from the topic, namespace, or broker level. If you want to use the initial configuration, please use the pulsar-admin
+  API orCLI:
+    - CLI
+      ```shell
+      bin/pulsar-admin topics set-replicated-subscription-status <tenant>/<namespace>/<topic> -s <subName> --disable
+      ```
+    - API
+      ```java
+      admin.topics().setReplicatedSubscriptionStatus(topic, subName, false);
+      ```
 
 ## Downgrade / Rollback
 
-If the broker is downgrade, users will need to ensure that the subscription replication configuration is re-enabled to maintain
-the correct replication behavior by the admin CLI or API:
-- CLI:
+If the broker is downgrade, users need to ensure that the subscription replication configuration is reset to maintain
+the correct replication behavior by the pulsar-admin CLI or API:
+- CLI
   ```shell
-  pulsar-admin topics set-replicated-subscription-status <tenant>/<namespace>/<topic> -s <subName> -e
+  # enable
+  pulsar-admin topics set-replicated-subscription-status <tenant>/<namespace>/<topic> -s <subName> --enable
+  # disable
+  pulsar-admin topics set-replicated-subscription-status <tenant>/<namespace>/<topic> -s <subName> --disable
   ```
-- API:
+- API
   ```java
+  // enable
   admin.topics().setReplicatedSubscriptionStatus(topic, subName, true);
+  // disable
+  admin.topics().setReplicatedSubscriptionStatus(topic, subName, false);
   ```
 
 ## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations