ACKs handling in sending multiple files

[TanmayRajore]

I am trying to send multiple files using the msquic and for each file, a new stream is being created and at the receiver side I print the filename once received but I receive all the files in one go so the order is not maintained, while debugging i found that the streamcallback event QUIC_STREAM_EVENT_SEND_COMPLETE for each file occurs only after all the files have been sent. Ideally, the working should be after each file sent the quic_Stream_event_send_complete should run and clear the stream(trigger for file sent complete on receiver side) but it all happens only at the end for all the streams opened. Is there a way to change this?

[nibanks]

I receive all the files in one go so the order is not maintained

Not sure but it seems you're saying that streams created in a particular order don't necessarily get received in that same order. Each stream is independent of one another. A primary goal/design of QUIC is that one stream shouldn't delay another.

event QUIC_STREAM_EVENT_SEND_COMPLETE for each file occurs only after all the files have been sent

My guess is that you're using send buffering (it's on by default). Which means MsQuic copies the data locally (just in memory) and then completes your send. If you want the send to be completed only after the peer (QUIC stack, not app) has received the data, then you should turn off send buffering (btw, this is a more performant design too, but has caveats). Please see SendBufferingEnabled.

[murphypei]

Can you collect the logs for a minimal repro and share them?

I think I also encountered a similar problem. I want to send multiple times in the same stream, if I wait in different sending, I can only get QUIC_STREAM_EVENT_SEND_COMPLETE event after all sending and the server can't receive any data.

the client send:

//
    // Starts the bidirectional stream. By default, the peer is not notified of
    // the stream being started until data is sent on the stream.
    //
    if (QUIC_FAILED(Status = MsQuic->StreamStart(Stream, QUIC_STREAM_START_FLAG_IMMEDIATE))) {
        printf("StreamStart failed, 0x%x!\n", Status);
        MsQuic->StreamClose(Stream);
        goto Error;
    }
   
    for (int i = 0; i < 10; ++i) {
        
        sleep(1);
        //
        // Allocates and builds the buffer to send over the stream.
        //
        SendBufferRaw = (uint8_t*)malloc(sizeof(QUIC_BUFFER) + SendBufferLength);
        if (SendBufferRaw == NULL) {
            printf("SendBuffer allocation failed!\n");
            Status = QUIC_STATUS_OUT_OF_MEMORY;
            goto Error;
        }
        SendBuffer = (QUIC_BUFFER*)SendBufferRaw;
        SendBuffer->Buffer = SendBufferRaw + sizeof(QUIC_BUFFER);
        SendBuffer->Length = SendBufferLength;

        gettimeofday(&time, NULL);
        printf("[strm][%p] Sending data...%ld\n", Stream, time.tv_sec * 1000000 + time.tv_usec);

        //
        // Sends the buffer over the stream. Note the FIN flag is passed along with
        // the buffer. This indicates this is the last buffer on the stream and the
        // the stream is shut down (in the send direction) immediately after.
        //
        if (QUIC_FAILED(Status = MsQuic->StreamSend(Stream, SendBuffer, 1, QUIC_SEND_FLAG_FIN, SendBuffer))) {
            printf("StreamSend failed, 0x%x!\n", Status);
            free(SendBufferRaw);
            goto Error;
        }
    }

the client log:

$ ./build/bin/Debug/quicsample -client -unsecure -target:localhost
[conn][0x55d582532210] Connecting...
[conn][0x55d582532210] Connected
[strm][0x7ff1000287c0] Starting...1668847251711208
[strm][0x7ff1000287c0] Sending data...1668847252711376
[strm][0x7ff1000287c0] Sending data...1668847253711729
[strm][0x7ff1000287c0] Sending data...1668847254712070
[strm][0x7ff1000287c0] Sending data...1668847255712437
[strm][0x7ff1000287c0] Sending data...1668847256712608
[strm][0x7ff1000287c0] Sending data...1668847257712804
[strm][0x7ff1000287c0] Sending data...1668847258713021
[strm][0x7ff1000287c0] Sending data...1668847259713244
[strm][0x7ff1000287c0] Sending data...1668847260713461
[strm][0x7ff1000287c0] Sending data...1668847261713793
[strm][0x7ff1000287c0] Data sent: 1668847261714053
[strm][0x7ff1000287c0] Data sent: 1668847261714074
[strm][0x7ff1000287c0] Data sent: 1668847261714091
[strm][0x7ff1000287c0] Data sent: 1668847261714102
[strm][0x7ff1000287c0] Data sent: 1668847261714113
[strm][0x7ff1000287c0] Data sent: 1668847261714127
[strm][0x7ff1000287c0] Data sent: 1668847261714138
[strm][0x7ff1000287c0] Data sent: 1668847261714151
[strm][0x7ff1000287c0] Data sent: 1668847261714269
[conn][0x55d582532210] Shut down by transport, 0x7d
[strm][0x7ff1000287c0] Data sent: 1668847261716142
[strm][0x7ff1000287c0] All done
[conn][0x55d582532210] All done

the server log:

Press Enter to exit.

[conn][0x7fbee0000b60] Connected
[strm][0x7fbee001d5e0] Peer started 1668847261715062
[conn][0x7fbee0000b60] Successfully shut down on idle.
[strm][0x7fbee001d5e0] All done
[conn][0x7fbee0000b60] All done

[TanmayRajore]

using the sample.c

server side :

void call_to_stream(char s1[],_In_ HQUIC Connection)
{
    QUIC_STATUS Status;
    HQUIC Stream;
 Status = MsQuic->StreamOpen(Connection, QUIC_STREAM_OPEN_FLAG_NONE, ServerStreamCallback, NULL, &Stream);
 ServerSend(Stream,s1); // stream send on with file as s1 and on the new stream
 usleep(500*1000);
// printf("%d,%p",Status,Stream);


}

for (int i = 0; i < 36; ++i) {

s1=file path
call_to_stream(s1,connection)
}

on the client side : I just store the stream buffers into a file and report the file as received when stream_send_Event_complete executes
but as reported above, the event occurs only after all the files have been sent from the server side. that is, the for loop completes.

server terminal output:
server_logs.txt
client terminal output:
client_logs.txt

Also currently I have set the idle timeout to indefinite otherwise the connection keeps getting shut down by transport if the data takes longer than say the default 1 second, is there a better way to handle this?
I will try and also provide the trace logs

Thanks

[nibanks]

If this is on Windows, please collect and share the ETL logs from Diagnostics.md with Full.Light profile ideally, but if that's too big, Datapath.Light instead.

[TanmayRajore]

quic_logs.zip
here are the logs

[nibanks]

I need the .etl please, not the .log file.