diff --git a/pip/pip-263.md b/pip/pip-263.md new file mode 100644 index 0000000000000..fad2c64ff1991 --- /dev/null +++ b/pip/pip-263.md @@ -0,0 +1,69 @@ +# Background knowledge & Motivation + +- Discussion: https://lists.apache.org/thread/tp25lom2ggztljlo76krsldo270f293j +- Vote: https://lists.apache.org/thread/10931v20ryr9m08qrl1yn8g6zlqfb4nc + +#### Just auto-create no-partitioned DLQ/Retry Topic + +If enabled the config `allowAutoTopicCreation,` Pulsar will auto-create a topic when the client loads it; After setting config `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, Pulsar will auto-create a partitioned topic(which have two partitions) when the client loads it. + +After the above, if using the feature [Retry Topic](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic) and [DLQ](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic) enable topic auto-creation, we will get a partitioned DLQ and a partitioned Retry Topic like this: +- `{primary_topic_name}-{sub_name}-DLQ` + -`{primary_topic_name}-{sub_name}-DLQ-partition-0` + -`{primary_topic_name}-{sub_name}-DLQ-partition-1` +- `{primary_topic_name}-{sub_name}-RETRY` + -`{primary_topic_name}-{sub_name}-RETRY-partition-0` + -`{primary_topic_name}-{sub_name}-RETRY-partition-1` + +---- + +I feel that almost all users will not use the multi-partitioned DLQ or multi-partitioned Retry topic because there is a bug that causes the above behavior to be incorrect, but we have yet to receive any issues about it. This bug causes the above behavior to look like this: When the partitioned DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will create a partitioned topic meta which has two partitioned but only create a topic named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug description. + +So I want to change the behavior to Just auto-create no-partitioned DLQ/Retry Topic. + + +---- + +#### Prevent auto-create the DLQ for a DLQ +If we use regex-topic(not filter out the Retry topics) consumer and enable retry, and after several times restart the client. it is possible to create such a topic `persistent://public/default/tp1-sub1-RETRY-sub2-RETRY-sub3-RETRY....`. Please look at this [Discussion](https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for the detail. + +# Goals + +- Just auto-create no-partitioned DLQ/Retry Topic(with the other words: prevent auto-create partitioned DLQ) +- DLQ/Retry topic should not create for a DLQ/Retry Topic + - roles: + - DLQ will not auto-create for a DLQ + - Retry Topic will not auto-create for a Retry Topic + - DLQ will not auto-create for a Retry Topic + - Retry Topic will not auto-create for a DLQ + - client changes: Clients will not create a DLQ for a DLQ + - broker changes: rejected the request which wants to auto-create a DLQ for a DLQ + +# Detailed Design + +**Properties of Topic** +```properties +"wasAutoCreated": boolean +``` + +- If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned topics will be automatically created +- If the property `wasAutoCreated` of the topic is true and the topic name ends with "-RETRY" or "-DLQ", retry topics and dead letter queues will no longer be created for this topic. + +# Backward & Forward Compatability + +## Revert + +Nothing todo. + +## Upgrade + +This change and PR https://github.com/apache/pulsar/pull/19841 will be released in the same version. The bug fixed by +https://github.com/apache/pulsar/pull/19841 maybe cause data loss of Retry topic and DLQ. + +The identification of the bug +- `allowAutoTopicCreation = true` +- `allowAutoTopicCreationType = partitioned` +- there has any ZK node(partitioned topic) named `*-partition-*` under the path `/admin/partitioned-topics`, but there has no related partition under the path `/managed-ledgers/{tenant}/{namespace}/persistent/` + +### Workaround +delete the ZK node whose name matches `*-partition-*` under the path `/admin/partitioned-topics`