Documenting channels with late or out-of-sequence events

[dalelane]

or "How do we describe time in asynchronous APIs?"

When messages are sent to a channel from multiple sources, it is common for the resulting messages to be slightly out of sequence. This can be because the different producers each have system clocks that are slightly out of sync, or because they take slightly different amounts to time to generate and/or send the events to the channel.

In such situations, developers who create a receiver to perform time-sensitive processing of this stream of messages need to take this into account.

There are ways that they can correctly handle this. Stream processing libraries, frameworks and applications (such as Apache Flink, Apache Spark, Kafka Streams, etc.) characterise this requirement in different ways - but often as handling “late” events or handling events out of sequence. Regardless of the specific technology, the common factor is that to be able to correctly process the stream of messages the consumer needs to know that they will need to handle this, and they need to know the degree of lateness they should expect.

We propose making this a property of channels, because it is a characteristic of the sequence of messages, not of any individual message.

One possible implementation of documenting this could look like this:

components:
  messages:
    transaction:
      payload:
        properties:
          id:
            type: string
            format: uuid
            description: unique id of a financial transaction
          accountnum:
            type: string
            description: identifier of the account that the transaction applied to
          txnvalue:
            type: number
            description: value of the transaction in some currency
          txntimestamp:
            type: string
            format: date-time
            description: official time that the financial transaction applied

  channels:
    accountTransactions:
      messages:
        transaction:
          $ref: '#/components/messages/transaction'
      time:
        # Maximum lateness for events on the channel,
        #  in milliseconds
        maxLateness: 180000

        # What is the canonical time for each message?
        #  In the message metadata? Or in the payload?
        #  If not specified, a reasonable default would
        #   be to assume the metadata holds the
        #   reference time
        location: payload
        # Which property holds the canonical time for
        #  each message? (applicable when the timestamp
        #  is in the payload)
        attribute: txntimestamp

Expressing the degree to which events can be out of sequence as “lateness” is a convenient way to put a numeric value to it, and is consistent with the way that it is described in many common stream processing technologies.

Putting this as a property of a “time” object will allow other time-based characteristics of the channel to be documented.

The example shows other values that it could be useful to document, such as how to identify whether the time for a given event should be identified from a property in the payload or in the envelope or metadata.

For developers of message senders, documenting lateness is a contract - a design constraint and commitment to delivering events from all sources within the time they specify.

For developers of message receivers, the lateness will let them to configure the framework or library that they use to correctly process events.

We are looking for feedback on how and where to best document this characteristic of asynchronous APIs.

[Anuragsharma15-dell]

hey could you assign this to me

[AtharvGhumtane]

Hi,
I’m interested in working on this issue. I’d like to contribute by helping draft a clear spec / documentation section that explains how late or out-of-sequence events can be described at the channel level, and include some example YAML based on the current proposal.
Please let me know if that sounds reasonable.

[Anuragsharma15-dell]

Document event-time semantics as an explicit, first-class property of the channel in an asynchronous API specification.
Concretely:
Add a time (or eventTime) object at the channel level
This captures properties of the stream as a whole, not individual messages.
Specify three key attributes
Canonical time source
Where event time is found (payload, headers, or metadata)
Which attribute contains it (e.g., txntimestamp)
Maximum expected out-of-orderness
Expressed as maxLateness (e.g., milliseconds or ISO-8601 duration)
Semantic meaning
Clarify that this is event time, not processing or ingestion time
Use lateness as the standard abstraction
Aligns with common stream-processing models (Flink, Spark, Kafka Streams)
Allows consumers to configure watermarks, windows, and buffering correctly
Treat it as a contract
Producers commit to delivering events within the stated lateness bound
Consumers rely on this to safely handle reordering and late arrivals
Outcome
This approach:
Makes time semantics explicit and discoverable
Keeps message schemas clean
Is portable across streaming technologies
Provides clear operational expectations for both producers and consumers I think with this approach we can go please tell me if it is wrong

[dalelane]

@Anuragsharma15-dell @AtharvGhumtane Thanks for the offer to get involved.

My issue is tagged as RFC0 - you can see a little context about what that means at https://github.com/asyncapi/spec/blob/master/CONTRIBUTING.md#rfc-contribution-stages but essentially at this point I'm looking for opinions.

I'm looking for people who work in event driven systems to share their experiences about how relevant and useful (or not!) they think such an extension would be.

I've provided one perspective - from my background with using Kafka Streams and Apache Flink, and from my limited knowledge of Spark. My perspective comes from working with IBM customers on Kafka projects - that's a narrow perspective, that doesn't reflect every type of asynchronous use case. The value we get from the AsyncAPI community coming together to discuss ideas is that we can draw on a broader set of perspectives before we settle on an idea.

Maybe you'd think this isn't a requirement you've seen before.
Maybe you've needed some of what I've suggested, but not all of it.
Maybe you'd need additional properties I haven't thought to include.
Maybe you've used a different application framework I'm unfamiliar with what would need time and lateness expressed in a different way.
Maybe you think there are other ways to express it that would be easier for the event source / producer to reasonably know.

These are the sorts of opinions and perspectives I would very much welcome.

I feel like so far you're mostly echo'ing back to me what I've already said. It's a good start to confirm you understand what I'm proposing - but for the next step, please take that away, think about it, and then come back with some ideas and opinions.

Tell me what I've missed. Tell me why I'm wrong. Tell me why it's too complicated or difficult. Tell me what this wouldn't work.

There must be something... there is no chance I've come up with something perfect on my first try ;-)

[Anuragsharma15-dell]

Oh thanks for the clarification but its really nice

[AtharvGhumtane]

Thanks for the detailed clarification, that helps a lot.

Thinking about this from my (more limited) experience, mainly coursework and a few small Kafka-based projects, I do see out-of-order events as a real problem — but I’m less sure how often producers can realistically commit to a concrete “max lateness” value.

One concern I have is that in many setups the producer doesn’t fully control delays once messages leave the application (network, brokers, retries, backpressure), so documenting lateness as a strict contract might be difficult to guarantee in practice. It feels like something consumers or stream processors reason about more than producers.

I also wonder whether this is somewhat framework-dependent. For example, systems like Flink have strong event-time semantics and watermarks, but in simpler pub/sub systems or IoT-style messaging, timestamps are often best-effort and sometimes unreliable or even missing. In those cases, I’m not sure how useful a channel-level time contract would be.

Another question I had is whether channel-level is always the right scope. If a channel carries multiple message types with different characteristics, could they have different lateness expectations, or would that add too much complexity?

Overall, I think the idea is useful, especially for stream-processing-heavy use cases, but I’m not yet convinced it generalizes cleanly to all async APIs. I’d be interested to hear how others with non-streaming or IoT workloads see this.

Happy to think more about this and iterate.

[dalelane]

Thanks for engaging with this - the debate is useful.

I’m less sure how often producers can realistically commit to a concrete “max lateness” value

Imagine I am the owner of the event source - I run the app that is producing events to the topic.

If I know that I only run one instance of the application, and perhaps even use transactions to fence out zombie producers, then I could confidently assert that there will not be any out of sequence events.

If I know that, I should (in my opinion) have a way to notify consumers of this.

If I'm running two instances of the application, I do some back-of-the-envelope maths to estimate the possible race condition between them - I calculate how much further one instance could get ahead of the other in how they source and produce events. I still add some buffer to this estimate, to allow for extreme cases - the amount of buffer depends on my level of confidence in my initial estimate.

I expect it can only ever be an estimate - at the point where you're able to precisely specify exactly how late your event will be, you will likely have enough control to avoid it being late in the first place. I was trying to use "max" to describe an upper limit for such an estimate, but perhaps the fact that I specified it in milliseconds implied a greater level of precision than I was intending. Perhaps describing it more overtly as an order-of-magnitude (e.g. milliseconds vs seconds vs minutes vs hours) would more clearly explain the intent.

One concern I have is that in many setups the producer doesn’t fully control delays once messages leave the application (network, brokers, retries, backpressure), so documenting lateness as a strict contract might be difficult to guarantee in practice. It feels like something consumers or stream processors reason about more than producers.

Ultimately, this information is required (literally) to be able to do stateful, time-based processing of events. I'm not sure who would be better able to provide a guide to this info than the owner of the application(s) that is producing the events.

In my opinion, it seems unreasonable to expect someone creating an application to consume an existing stream of events to know details of how they are produced. I feel like the place they should go to learn about this is, almost by definition, the AsyncAPI doc.

I also wonder whether this is somewhat framework-dependent. For example, systems like Flink have strong event-time semantics and watermarks, but in simpler pub/sub systems or IoT-style messaging, timestamps are often best-effort and sometimes unreliable or even missing.

Sorry, you lost me a little here.

When you say that "Flink has strong event-time semantics and watermarks" - that is true... but what that means is that Flink has robust ways to let you reason about time in your processing. It doesn't mean that Flink is injecting the time - that comes from the event source (in AsyncAPI terms, the channel that Flink is consuming from).

Flink isn't a source of events - it's the runtime you use to process the events. If you use it to process events from SQS, the source of time is SQS. If you use it to process events from Kafka, the source of time is Kafka. If you use it to process events from RabbitMQ, the source of time is RabbitMQ. If you're using Flink to process events from a pub-sub system or an IoT-style system, it will use the timestamps it gets from those systems. And so on.

If you have reason to think that an IoT platform that is hosting a source of events will be likely to introduce some additional level of out-of-sequence-ness in the timestamps, then I am arguing that someone consuming those events (whether with Flink, or with anything else) could benefit from being notified about that. That is the sort of thing I am describing above.

Another question I had is whether channel-level is always the right scope. If a channel carries multiple message types with different characteristics, could they have different lateness expectations, or would that add too much complexity?

Lateness isn't an attribute of any single event in isolation. It is a description of the relative time difference between different events on the topic. So that's why I was thinking that the channel was the best place for it.

Overall, I think the idea is useful, especially for stream-processing-heavy use cases, but I’m not yet convinced it generalizes cleanly to all async APIs. I’d be interested to hear how others with non-streaming or IoT workloads see this.

I agree with the statement that 100% of use cases will not require this. My experience would suggest that IoT use cases aren't necessarily any less likely than other domains, but irregardless - yes, this isn't something I think should be a mandatory part of every AsyncAPI document. But I don't feel like that should be a requirement - there are several optional fields that similarly aren't useful in 100% of use cases.

I think it's reasonable for us to explore whether it's a widespread requirement, but I think needing it to be applicable to all AsyncAPIs is setting the bar a little too high.

Thanks again for the input

[AtharvGhumtane]

Thanks for taking the time to explain this in detail — that definitely clarified a few things for me.

I agree with your point that the owner of the event source is in the best position to provide this information, and that expecting consumers to infer production characteristics would be unreasonable. Framed that way, documenting lateness as part of the AsyncAPI contract makes a lot more sense to me.

The distinction you made about “max” being an upper bound or order-of-magnitude estimate, rather than a precise guarantee, also helped. I think my hesitation was partly driven by the millisecond-level precision in the example, which initially made it feel more exact than what producers could realistically promise.

One thought that might help avoid that interpretation is whether the spec should be very explicit (in wording or examples) that this value is best-effort guidance rather than a strict SLA. Alternatively, leaning into coarser or descriptive guidance (e.g. order-of-magnitude, or clearly stating the assumptions behind the estimate) could make it easier for producers to provide and for consumers to interpret correctly.

I also take your point on Flink — that the semantics live in the processing framework, but the timestamps and their quality ultimately come from the source. Seen that way, documenting potential out-of-sequence behaviour at the channel level feels reasonable.

Overall, I’m more convinced now about the usefulness of this for stateful, time-based processing, even if it remains optional. I think the key will be making the intent and limitations of the field very clear so it’s used as guidance, not over-trusted.

Thanks again — this has been a really useful discussion.

[derberg]

regarding location

        # What is the canonical time for each message?
        #  In the message metadata? Or in the payload?
        #  If not specified, a reasonable default would
        #   be to assume the metadata holds the
        #   reference time
        location: payload
        # Which property holds the canonical time for
        #  each message? (applicable when the timestamp
        #  is in the payload)
        attribute: txntimestamp

imho better would be to use the mechanism we already know and use in other places of the spec, which is runtime expression

"location": "$message.payload#/txntimestamp"

other places in spec where it is used: https://www.asyncapi.com/docs/reference/specification/v3.0.0#correlationIdObject

regarding streaming

if this is strictly streaming related, woulnd't it be better to have a dedicated object, streaming where in the future other props could be added? watermark?

it's not protocol-related, so not for bindings.