Is there any way to take ownership of received buffer memory *without* pausing the stream?

[jeremyong]

Suppose you receive a set of buffers on a stream's receive event and want to do further processing asynchronously. You aren't constrained or congested with respect to machine resources, so you don't want the connection to stop receiving additional data (at any cost). However, you also don't want to copy the buffer memory, preferring to have ownership of it until the async processing is done. What's the idiomatic way of achieving this with msquic? At least based on documentation, it seems like the only way to do this is to return a pending status to the receive indicator and then immediately re-enable the stream?

edit: to clarify, the "async processing" to be done on received data is some form of I/O that doesn't occupy a core that could otherwise be used for packet processing or other tasks

[jeremyong]

Looking at

msquic/src/core/stream_recv.c

Line 750 in f1b6996

QUIC_STATUS Status = QuicStreamIndicateEvent(Stream, &Event);

specifically, it seems like what I might want is a status flag which sets the stream flag ReceiveCallPending without setting ReceiveEnabled to false. However, I'm not sure what other interactions there may be that make this possible or not possible.

[nibanks]

I would really recommend starting with the current async model we have of:

• Return QUIC_STATUS_PENDING
• Process the buffer(s) on your own thread
• Complete the send via StreamReceiveComplete

This doesn't mean we stop processing the stream at the QUIC layer. We'll continue to buffer the received data (up to the current FC window), but just won't give you a new callback until you're done with what you already have. If you don't think that would work, could you explain your scenario a bit more to explain why not?

[jeremyong]

One other use case is suppose you want to defer the decision making of what to do with the packet until a future control signal, but the server has sent you the data preemptively.

[nibanks]

Your second case actually has a chance to deadlock. If you (the app) don't drain the receive buffer, but instead just wait for more, AND there isn't more space because of flow control, then you're stuck. So generally, don't use the the transport as temporary storage while you wait for more, unless you can guarantee there is enough space for what you're waiting. BTW, if you try to do the same kind of blocking and waiting across streams, then things get even more complicated and you have even more chance of deadlock.

As for your first, if you're going to do some other IO with the received data, then I'd recommend you either copy or just use the current async model, and measure the results. I'm hesitant to add yet more complexity in this area (it's already a bit lax on test coverage as it is) without data backing up the need to make a change.

[jeremyong]

Maybe I can frame this as a more general question then (trying to think of the simplest use case that might need this). What would, in your mind, be the best way to download data via msquic and save it to disk? Would you do intermediate copies of the buffers to page size buffers that flush to disk when they get full? Would you return "pending" and complete after the transport buffer was flushed to disk?

Thanks for the responses btw

[nibanks]

I think I'd use the PENDING model, and pass the buffer to the disk IO call. One thing that might help is a SetParam that allows you to explicitly set your stream's FC window.

[jeremyong]

Thanks I'll give that a try. It occurred to me that for this particular use case, the ideal API would actually be one in which you can provide msquic backing memory upfront for a stream in the event the size of the payload is known. This also solves issues where alignment is an issue, for example, when filling data you intend to stream directly to the GPU, or if you intend on writing directly to disk via unbuffered writes (so the memory must be aligned to the physical or logical sector size).