Explicitly producing/consuming to/from a specific topic partition

[SantanuKar43]

What is the reason behind allowing pulsar client to directly produce or consume from specific partitions by appending: "-partition-[n]" in the topic name? Shouldn't this be abstracted out from the client? Is it an anti-pattern to explicitly specify partitions?

[lhotari]

Pulsar allows direct partition access (via -partition-[n] suffix) because of how it's architected internally. Under the hood, partitioned topics are actually separate non-partitioned topics with a naming convention. On the broker side, the partitioned topic itself is only metadata about how many partitions there are in total.

This gives developers flexibility for advanced scenarios but doesn't mean you should always use it. For most use cases, you're better off using the higher-level abstractions provided by the client libraries.

The client handles partition management internally through components like MultiTopicsConsumerImpl and PartitionedProducerImpl, so you typically don't need to worry about specific partitions.

I wouldn't call direct partition access an anti-pattern, but it's definitely a lower-level approach that should be used only when you have specific needs that can't be met through the standard abstractions. Administrative use cases for inspecting individual partitions and certain strict ordering scenarios are examples where it's necessary to address individual partitions directly in applications. Another example could be an efficient solution to the usecase described in #24044.