The producer is gone, after modifying the parameter defaultNumberOfNamespaceBundles and restarting the brokers

[dsmlily]

Expected behavior

The application can continue to work without being affected

Actual behavior

the application cannot produce messages to the broker

Steps to reproduce

Occasional

System configuration

Pulsar version: 2.4.0
There are 3 brokers in our Pulsar cluster。

What happened

The parameter defaultNumberOfNamespaceBundles was set 4. After Modify parameter defaultNumberOfNamespaceBundles to 6 and restarting the brokers, the producers of one topic were gone. Every partition topic should have two producers.

before restarting the brokers :
time="2020-02-17T05:00:29+08:00" level=debug msg="Lookup result: &{LogicalAddr:pulsar://10.32.69.26:6651 PhysicalAddr:pulsar://10.32.69.26:6651}" name=qtvhr subscription=worker_1v1 topic=2/signal_geo/redis_sync-partition-0
time="2020-02-17T04:59:56+08:00" level=debug msg="Lookup result: &{LogicalAddr:pulsar://10.32.69.26:6650 PhysicalAddr:pulsar://10.32.69.26:6650}" name=qtvhr subscription=worker_1v1 topic=2/signal_geo/redis_sync-partition-1

after restarting the brokers:
[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics lookup persistent://2/signal_geo/redis_sync-partition-0
"pulsar://10.32.69.26:6650"
[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics lookup persistent://2/signal_geo/redis_sync-partition-1
"pulsar://10.32.69.26:6652"

[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics stats persistent://2/signal_geo/redis_sync-partition-0
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 103,
"publishers" : [ ],
"subscriptions" : {
"worker_1v1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "auurc",
"availablePermits" : 2048,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"address" : "/10.32.32.223:41824",
"connectedSince" : "2020-02-21T20:25:16.856+08:00"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "lnmnh",
"availablePermits" : 2048,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"address" : "/10.32.68.186:30702",
"connectedSince" : "2020-02-21T20:25:16.857+08:00"
} ],
"isReplicated" : false
}
},
"replication" : {
"qa-pulsar-ten" : {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 0,
"connected" : true,
"replicationDelayInSeconds" : 0,
"inboundConnection" : "/10.32.32.20:5644",
"inboundConnectedSince" : "2020-02-24T18:54:16.203+08:00",
"outboundConnection" : "[id: 0x4fd18011, L:/10.32.69.26:34288 - R:10.32.32.20/10.32.32.20:6651]",
"outboundConnectedSince" : "2020-02-24T18:54:16.257+08:00"
}
},
"deduplicationStatus" : "Disabled"
}

[duansumin@l-middleware-server1.sig.test.ten.dm apache-pulsar-ali]$ ./bin/pulsar-admin topics stats persistent://2/signal_geo/redis_sync-partition-1
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 590,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"producerId" : 3,
"metadata" : { },
"producerName" : "qa-pulsar-ali-29-4286",
"connectedSince" : "2020-02-21T20:24:43.622+08:00",
"address" : "/10.32.68.186:43841"
} ],
"subscriptions" : {
"worker_1v1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "lnmnh",
"availablePermits" : 3061,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2020-02-21T20:25:16.647+08:00",
"address" : "/10.32.68.186:43841"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "auurc",
"availablePermits" : 3072,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2020-02-21T20:25:16.751+08:00",
"address" : "/10.32.32.223:49520"
} ],
"isReplicated" : false
}
},
"replication" : {
"qa-pulsar-ten" : {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateExpired" : 0.0,
"replicationBacklog" : 0,
"connected" : true,
"replicationDelayInSeconds" : 0,
"inboundConnection" : "/10.32.32.20:7870",
"inboundConnectedSince" : "2020-02-24T19:40:16.119+08:00",
"outboundConnection" : "[id: 0xbba2928c, L:/10.32.69.26:4291 - R:10.32.32.20/10.32.32.20:6652]",
"outboundConnectedSince" : "2020-02-24T19:40:16.153+08:00"
}
},
"deduplicationStatus" : "Disabled"
}

[sijie]

@dsmlily do you mean the producer stops producing messages?

[dsmlily]

not only the producer stops producing messages but also there is not the producer in the client lib