[improve][broker] PIP-442: Add memory limits for topic list watcher (part 2)

[lhotari]

PIP-442

Motivation

See PIP-442.

This is part 2 for the PIP-442 implementation. This covers the topic list watcher operations.
The first part of PIP-442 implementation was #24833.

This PR implements memory limits and flow control for the CommandWatchTopicList feature as specified in PIP-442. Currently, topic list watcher operations can allocate unbounded amounts of memory based on namespace size, creating the final significant gap in Pulsar's otherwise comprehensive memory management system.

The lack of memory limits creates several critical issues:

• Broker Memory Exhaustion Risk: Large namespaces can trigger OutOfMemoryErrors when multiple clients establish topic list watchers simultaneously
• Unpredictable Resource Usage: Operators cannot reliably predict or limit total broker memory consumption due to this unbounded allocation path
• Performance Degradation: Even without OOM, large topic list operations cause GC pressure and latency spikes affecting all broker operations

This PR completes Pulsar's memory management framework by implementing memory-based flow control and backpressure specifically for topic list watcher operations.

Modifications

• use memory limits in TopicListService / PulsarCommandSenderImpl so that topic list watcher related operations are covered
• Modified PulsarCommandSender interface: Changed sendWatchTopicListSuccess and sendWatchTopicListUpdate methods to return CompletableFuture<Void> and accept permit acquire error handlers for asynchronous memory management
• Enhanced TopicListService:
  • Added permit-based memory control with retry logic for sending topic list responses
  • Implemented backoff strategy for permit acquisition with configurable timeouts and queue sizes
  • Added proper cleanup and cancellation support when watcher operations fail or are closed
• Added comprehensive testing: Created PatternConsumerTopicWatcherBackPressureMultipleConsumersTest to verify backpressure behavior with multiple concurrent consumers
• Updated existing tests: Modified TopicListServiceTest and TopicListWatcherTest to accommodate the new asynchronous response pattern

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

❌ Patch coverage is 69.15888% with 66 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.55%. Comparing base (413387b) to head (e41fc18).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...apache/pulsar/broker/service/TopicListService.java	70.20%	37 Missing and 22 partials ⚠️
...pulsar/broker/service/PulsarCommandSenderImpl.java	33.33%	0 Missing and 2 partials ⚠️
...r/common/semaphore/AsyncDualMemoryLimiterImpl.java	0.00%	2 Missing ⚠️
...va/org/apache/pulsar/broker/service/ServerCnx.java	0.00%	1 Missing ⚠️
...he/pulsar/common/semaphore/AsyncSemaphoreImpl.java	0.00%	1 Missing ⚠️
...apache/pulsar/proxy/server/LookupProxyHandler.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #25070       +/-   ##
=============================================
+ Coverage     37.48%   72.55%   +35.07%     
- Complexity    13220    34266    +21046     
=============================================
  Files          1899     1956       +57     
  Lines        150586   154897     +4311     
  Branches      17159    17661      +502     
=============================================
+ Hits          56440   112392    +55952     
+ Misses        86464    33505    -52959     
- Partials       7682     9000     +1318     

Flag	Coverage Δ
inttests	25.63% <2.80%> (-0.02%)	⬇️
systests	22.44% <6.07%> (+<0.01%)	⬆️
unittests	73.51% <69.15%> (+39.35%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...va/org/apache/pulsar/common/protocol/Commands.java	90.99% <ø> (+25.00%)	⬆️
...r/common/semaphore/AsyncDualMemoryLimiterUtil.java	100.00% <100.00%> (+39.28%)	⬆️
...ava/org/apache/pulsar/common/topics/TopicList.java	100.00% <100.00%> (+21.05%)	⬆️
...va/org/apache/pulsar/broker/service/ServerCnx.java	71.56% <0.00%> (+26.57%)	⬆️
...he/pulsar/common/semaphore/AsyncSemaphoreImpl.java	91.21% <0.00%> (+39.51%)	⬆️
...apache/pulsar/proxy/server/LookupProxyHandler.java	58.08% <0.00%> (+19.85%)	⬆️
...pulsar/broker/service/PulsarCommandSenderImpl.java	85.71% <33.33%> (+24.57%)	⬆️
...r/common/semaphore/AsyncDualMemoryLimiterImpl.java	91.11% <0.00%> (+14.36%)	⬆️
...apache/pulsar/broker/service/TopicListService.java	71.02% <70.20%> (+27.95%)	⬆️

... and 1408 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Denovo1998]

Left some comments.

[Copilot]

Pull request overview

This PR implements memory limits for topic list watcher operations as part of PIP-442. It introduces backpressure mechanisms to prevent broker memory exhaustion when handling concurrent pattern consumer topic list requests by applying memory limiting to topic list watcher creation and update operations.

Key Changes

• Added memory limit acquisition with retry logic for topic list watcher initialization and updates
• Modified topic list watcher to queue updates during initialization to avoid race conditions
• Enhanced test coverage for memory limiting scenarios and concurrent consumer patterns

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java	Added toString method to SemaphorePermit for debugging
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterImpl.java	Added toString method to DualMemoryLimiterPermit for debugging
pulsar-broker/src/test/java/org/apache/pulsar/client/api/PatternConsumerTopicWatcherBackPressureMultipleConsumersTest.java	New test validating backpressure with 100 concurrent pattern consumers and 300 topics
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicListWatcherTest.java	Updated test setup to use direct executor and verify watcher initialization completion
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicListServiceTest.java	Added comprehensive tests for permit acquisition retries and memory limiting scenarios
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java	Implemented memory limiting with retry backoff for watcher operations and queued updates during initialization
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarCommandSenderImpl.java	Modified methods to acquire direct memory permits before sending topic list messages
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarCommandSender.java	Updated interface to return CompletableFuture and accept error handler for permit acquisition

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Denovo1998]

@lhotari It seems it will still be out of order.

[Denovo1998]

The problem of scrambling seems to have been solved for now.

[BewareMyPower]

Overall LGTM except for my comments