FIFO guarantees with Key_Shared subscriptions when scaling partitions and clearing backlog

[vishnumurthy-nd]

I am relatively new to Apache Pulsar and am experimenting to understand its behavior under some real-world scaling scenarios. I would really appreciate guidance to validate whether my understanding and design approach is correct.

Requirements / Constraints:

1. Multi-tenant system where tenants (smart meter providers) are dynamically onboarded.
2. Each tenant has devices ranging from 10 to 1M, and device count can increase over time.
3. Strict FIFO ordering per device (key) must never be violated, even during downtime or scaling events.
4. System must tolerate periods where producers are active but consumers are temporarily down, causing backlog accumulation.

Proposed Design:

1. One topic per tenant.
2. Topic is partitioned for throughput.
3. Messages are produced with deviceId as the message key.
4. Consumers use Key_Shared subscriptions to guarantee per-key ordering and allow consumer scalability.

Scenario I’m Trying to Understand

1. A tenant topic initially has N partitions (e.g., 2).
2. Producers continue writing data for existing deviceIds while no consumers are active, so backlog accumulates.
3. Due to higher ingress or future scale needs, the partition count is increased to N + M (e.g., from 2 → 4).
4. Later, consumers start (or scale up).

Questions:

1. After increasing the partition count
   • Will messages for existing deviceIds continue to be routed to their original partitions?
   • Or can existing keys be rehashed to newly added partitions?
2. When consumers start after downtime:
   • How does Pulsar merge backlog + live traffic while preserving strict FIFO per key?
   • Is backlog always drained in partition order before newer messages for the same key are delivered?
3. With Key_Shared subscriptions:
   • Is per-key ordering guaranteed even if consumers join after backlog has accumulated?
   • Are there any edge cases where unacked messages for a key can block or delay delivery of other keys?
4. From a design perspective:
   • Is increasing partition count on an existing partitioned topic compatible with strict FIFO per key?
   • Or is the recommended approach to fix partition count at topic creation time and scale only via consumers?

I want to ensure I am not misunderstanding any internal guarantees or relying on behavior that is not officially supported.

Thanks in advance for your help and for maintaining Pulsar

[lhotari]

After increasing the partition count

• Will messages for existing deviceIds continue to be routed to their original partitions?
• Or can existing keys be rehashed to newly added partitions?

By default, messages will get routed based on the hash of the key. After increasing the partition count, new messages for some of the existing keys will go to added partitions.

When consumers start after downtime:

• How does Pulsar merge backlog + live traffic while preserving strict FIFO per key?

There's explanation of Pulsar subscription type in https://pulsar.apache.org/docs/4.1.x/concepts-messaging/#subscription-types.
A consumer has a subscription to a topic in Pulsar. The exclusive, failover and key_shared subscriptions consume messages in FIFO order.

• Is backlog always drained in partition order before newer messages for the same key are delivered?

yes. however if new partitions have been added, there's no special support for this. It's similar to Kafka in this matter. There have been some thoughts around adding a new feature to Pulsar 5.0 to address this shortcoming.

With Key_Shared subscriptions:

• Is per-key ordering guaranteed even if consumers join after backlog has accumulated?
  yes. The guarantee is explained in https://pulsar.apache.org/docs/4.1.x/concepts-messaging/#preserving-order-of-message-delivery-by-key .

• Are there any edge cases where unacked messages for a key can block or delay delivery of other keys?

yes. when new consumers join, the key space is repartitioned across consumers to balance the load. Outstanding unacknowledged messages will cause delivery of reassigned keys until the outstanding messages have been acknowledged. This has been improved since Pulsar 4.0.0, the docs at https://pulsar.apache.org/docs/4.1.x/concepts-messaging/#preserving-order-of-message-delivery-by-key explain more. In Pulsar 4.0.0+ this is handled at a hash level and it doesn't block all message delivery when rehashing happens.
Delivery of other message keys will continue as long as the number of temporarily skipped messages stay under the amount configured with keySharedLookAheadMsgInReplayThresholdPerConsumer/keySharedLookAheadMsgInReplayThresholdPerSubscription (values are 2000/20000 by default). The reason for this is to avoid the state of the subscription growing without a boundary.

[lhotari]

Regarding strict FIFO order, I have some content about this in my Data Streaming Summit 2025 talk, "Mastering Key-Ordered Message Processing in Apache Pulsar", recording at https://www.youtube.com/watch?v=AkSGvYP4r88 .

[vishnumurthy-nd]

@lhotari Thanks. If possible can you share the insights about addressing strict FIFO while re-partitioning topic in Pulsar 5.0.

[lhotari]

If possible can you share the insights about addressing strict FIFO while re-partitioning topic in Pulsar 5.0.

@vishnumurthy-nd It's still a high-level idea, and I've been picking @merlimat's brain while we've had a few discussions about it. I suspect it would be somewhat similar to Pravega's elastic streams. Pravega's stream segments can automatically grow and shrink over time based on I/O load.

Pulsar already has the Key_Shared subscription, which is sufficient for use cases where the throughput of a single broker serving a partition is adequate. Unless there's significant I/O load, there isn't a need for partitions in Pulsar. It's also possible to use Key_Shared subscriptions over partitioned topics when a single partition isn't sufficient; however, changing the number of partitions would break key-ordered processing.

The limitation of current approaches is that at high scale, they still require thinking about partitions when this could be eliminated while addressing the key-ordered processing problem inherent in current partition-based approaches.

One way to think about the idea is to consider it as a solution that manages partitions under the covers, scaling up and down over time. The solution stores the necessary metadata so that keys can be consumed in order when this happens. There would be necessary protocol changes between producers, consumers, and brokers to enable this. The challenge is that this new type of topic would require new clients that support this feature. There could be a proxy approach for backwards compatibility with existing clients, though with certain limitations.

@merlimat Did I understand your vision correctly?

[lhotari]

Pulsar 5.0 thread on dev-mailing list: https://lists.apache.org/thread/3zclb8cq0dt95c8f1lh83qxyyj11m8pq