TinyUSB CDC Flow Control vs (Hardware) USB<->Serial Converter

[mws-rmain]

I'm having problems using the TinyUSB CDC device as a replacement for a hardware USB UART. I've found that communication begins fine, in both directions (Tx & Rx), but it seems the TinyUSB CDC implementation is missing a few features that cause ultimate communication failure.
I'm using a serial port monitor program ( https://www.eltima.com/products/serial-port-monitor/ ) to capture & compare communications from a standard application (bootloader app running on Windows 10) on one board that uses a CP2102 USB-Serial converter, compared to another board that uses a MCU running a TinyUSB CDC instance (which maps to a hardware UART in the MCU). The serial monitor highlights several differences early on (IOCTL_SERIAL_GET_/SET_ CHARS/HANDFLOW, UP & DOWN). The main difference looks like XonLimit=32768, XoffLimit=8192 in the IOCTL_SERIAL_GET_HANDFLOW, which is zero under TinyUSB.

Also, in IOCTL_SERIAL_SET_TIMEOUTS (UP) ReadTotalTimeoutConstant=300, WriteTotalTimeoutConstant=300.

After numerous exchanges involving the above IOCTLs, the session trace shows a significant difference. The CP2102 configuration sees a large block of IOCTLs (SERIAL_GET/SET TIMEOUTS, WAIT_MASK, BAUD_RATE, LINE_CONTROL, CHARS, etc.), which is missing in the TinyUSB/MCU session trace. I can supply the session traces if it helps, but the (non-free) program would be required to read them.

I can't be sure, but I suspect the issue is lack of XON/XOFF support in TinyUSB. Do you have any plans to add this, or could you point me to required info to implement? I know TinyUSB supports DTR/RTS for down-stream flow control, but I don't see any implementation for up-stream flow control (e.g., DSR/CTS). For that matter, I don't know how you'd go about 'stalling' USB transfers if the MCU UART FIFOs were full anyway. XON/XOFF is another way to handle up & down-stream flow control, so might (?) be easier to implement.

[mws-rmain]

Digging deeper - and using a different session decode window ('Line View') shows that the application writes a block of 3365 bytes, then polls for completion (see excerpt below)

[16/11/2021 08:51:10] 
1489 IRP_MJ_WRITE - Request transfers data from a client to a COM port (COM31) - 3365 bytes of 3365 
    STATUS_SUCCESS 
            c0 00 07 db dc 0c ff 00 00 00 b0 0c 00 00 00 00   À..ÛÜ.ÿ...°..... 
            00 00 00 00 00 00 00 00 00 00 f8 20 f4 3f f8 30   ..........ø ô?ø0 
...
            82 a0 dd 7c f2 87 99 06 0c 12 29 03 22 a0 db dd   ‚ Ý|ò‡™...)." ÛÝ 
            1d f0 00 00 c0                                    .ð..À            
---------------------------------------------------------------------------------- 
[16/11/2021 08:51:10] 
1491 IRP_MJ_DEVICE_CONTROL - Request operates a serial port (COM31) 
    STATUS_SUCCESS 
        IOCTL_SERIAL_GET_COMMSTATUS - Request returns information about the communication status of a COM port 
            Errors           - 0 
            HoldReasons      - 0 
            AmountInInQueue  - 0 
            AmountInOutQueue - 0 
            EofReceived      - 0 
            WaitForImmediate - 0 
---------------------------------------------------------------------------------- 
[16/11/2021 08:51:10] 
1493 IRP_MJ_DEVICE_CONTROL - Request operates a serial port (COM31) 
    STATUS_SUCCESS 
        IOCTL_SERIAL_GET_COMMSTATUS - Request returns information about the communication status of a COM port 
            Errors           - 0 
            HoldReasons      - 0 
            AmountInInQueue  - 0 
            AmountInOutQueue - 0 
            EofReceived      - 0 
            WaitForImmediate - 0 
---------------------------------------------------------------------------------- 
[16/11/2021 08:51:13] 
1495 IRP_MJ_READ - Transfers data from a COM port to a client (COM31) - 0 bytes of 1 
    STATUS_TIMEOUT 
---------------------------------------------------------------------------------- 

in the CP2102 session:

• GET_COMMSTATUS returns 457 bytes for 'AmountInOutQueue',
• IRP_MJ_READ returns 'STATUS_SUCCESS' (instead of timeout), then
• another GET_COMMSTATUS returns 'AmountInInQueue'=13, followed by a read of 13 bytes.

I definitely don't have a 4K buffer in the MCU (and later in the CP2102 sequence a 16K block transfer occurs), so unless USB parcels the block into significantly smaller packets (and then uses XON/XOFF or some protocol to throttle delivery), this may be the source of my problem.

[mws-rmain]

The CP2102 has the same issue (640 byte Tx buffer, and 576 byte Rx buffer), so clearly there is some mechanism for USB to parcel & throttle...
?

[mws-rmain]

I need to better understand USB protocol.
I have verified the application is initiating a 'large' (3.5KByte) transfer. The HOST driver or USB controller will be breaking this up into packets not exceeding the device max_packet_size (64 bytes in my case).
I believe what I need is for the TinyUSB CDC driver to NAK packets from the HOST when it's rx FIFO is full.
I believe this should be happening in function _prep_out_transaction() in cdc_device.c here:

static void _prep_out_transaction (cdcd_interface_t* p_cdc)
{
  uint8_t const rhport = TUD_OPT_RHPORT;
  uint16_t available = tu_fifo_remaining(&p_cdc->rx_ff);

  // Prepare for incoming data but only allow what we can store in the ring buffer.
  // TODO Actually we can still carry out the transfer, keeping count of received bytes
  // and slowly move it to the FIFO when read().
  // This pre-check reduces endpoint claiming
  TU_VERIFY(available >= sizeof(p_cdc->epout_buf), );

  // claim endpoint
  TU_VERIFY(usbd_edpt_claim(rhport, p_cdc->ep_out), );

  // fifo can be changed before endpoint is claimed
  available = tu_fifo_remaining(&p_cdc->rx_ff);

  if ( available >= sizeof(p_cdc->epout_buf) )
  {
    usbd_edpt_xfer(rhport, p_cdc->ep_out, p_cdc->epout_buf, sizeof(p_cdc->epout_buf));
  }else
  {
    // Release endpoint since we don't make any transfer
    usbd_edpt_release(rhport, p_cdc->ep_out);
  }
}

Clearly, the data is not copied if there is no room in the FIFO, but all that happens is the endpoint is released. Shouldn't a flag be set to ensure the packet just received is NAK'd so the host will retry? If so, how is this accomplished? Or is the NAK triggered elsewhere somehow?

[HiFiPhile]

From your description I think there are some problem about how did you integrate TinyUSB into your project, especially how did you handle hardware UART transmit when data received from USB host.

Without more information is hard to know what happened, please provide at least your MCU/moard, minimal source.

I believe what I need is for the TinyUSB CDC driver to NAK packets from the HOST when it's rx FIFO is full.

It's fully handled by TinyUSB, you could try cdc_msc example and compare data written and data echoed.

[mws-rmain]

Thanks for the response, @HiFiPhile.
I've implemented interrupt driven Tx & Rx with FIFOs for the UART. I will see about setting up a more controlled test so I can control the size of the initial bulk transfer (to date, I've only used either a terminal, or the 'bootload application' (which I have no control over)).

I believe if you check the applicable examples (cdc_msc, cdc_dual_ports), you'll see they implement a direct 'echo' with a 64 byte buffer. The main while loop is very tight, servicing tud_task() & cdc_task(), so I believe any received packet will be immediately echoed - there will never be a case where a second received packed should be NAK'd to force the host to retry.

I'll see if I can hack one of the examples to delay echo of received data for some period of time, which should allow the rx_ff to fill (64 bytes), and require USB flow control to kick in. As long as I can verify all data sent was (eventually) received, then TinyUSB is doing everything fine. If data is dropped, at least I'll have baseline code to easily illustrate the issue.

BTW - I'm currently working with a SAMD21J18A on a SAMD21XPlained board (eventually moving to a custom board with another SAMD21 MCU).

[mws-rmain]

I made a copy of the cdc_msc example into a directory called 'cdc_flow_ctl'. A modified main.c file is in the attached ZIP.
The ZIP also contains two txt files: 'flow_ctl_lrg.txt' (197 bytes), and 'flow_ctl_sm.txt' (41 bytes).

I programmed the standard cdc_msc example into my board, and verified I can open a serial terminal (TeraTerm v4.92) on the USB CDC port (ensure 'local echo' is OFF). Any characters typed are echoed back, and sending either file returns the same data.

Then programmed the cdc_flow_ctl example into the board.
Typing a few characters will echo them back after several seconds delay.
Sending the flow_ctl_sm.txt file will echo it back after a short delay.
Sending the flow_ctl_lrg.txt file will not echo ANYTHING back (something is hung/crashed/corrupted at this point).

flow_ctl.zip

[HiFiPhile]

there will never be a case where a second received packed should be NAK'd to force the host to retry.

It's not how USB bulk transfer works. The host always wait the previous packet is processed before sending the next.

I'll try to investigate what happened.

[mws-rmain]

My theory was based on the flow described below (citation in at bottom)

:

[mws-rmain]

So what I think would happen is:

• Host sends 64 Byte packet;
• Rx FIFO is empty, so packet is received (FIFO is now full), and ACK'd;
• Host sends next 64 Byte packet;
• Rx FIFO is still full (has not yet been read & echoed to outbound endpoint), so CDC device should (?) NAK the packet (may happen multiple times);
• Eventually first packet is echoed to outbound endpoint, emptying the Rx FIFO.
• On the next delivery attempt, the second packet is accepted into the Rx FIFO & ACK'd;
• ... etc...

[mws-rmain]

Isn't this this comment in your Issue #920 describing pretty much what I think is the root problem?

[HiFiPhile]

tud_cdc_available() return the number of available characters, so you should use

if ( tud_cdc_available() && time_to_echo )

instead of

if ( tud_cdc_available() & time_to_echo )

Echo is working as intended.

[HiFiPhile]

Isn't this this comment in your Issue #920 describing pretty much what I think is the root problem?

It's not a problem, just a possible enhancement. My suggestion is please learn more how USB & TinyUSB stack works then you can poking around the stack for real issue.

[mws-rmain]

Thanks, @HiFiPhile for pointing out my '&&' error.
I fixed that, and tried again. Still got odd results using Tera Term - sending the 'flow_ctl_lrg.txt' file resulted in me receiving the first line over & over (I terminated it eventually).
Then I downloaded HTerm (since you used it successfully), and it worked as expected.

I'll dig some more into what the difference might be, but Tera Term is definitely having issues with the loopback.

It still seems to me that TinyUSB CDC is not sending a NAK when there is no buffer space available - if it is, I can't locate how/where. From the 'Bulk Transfer Error Control Flow' image I posted, I think TinyUSB might just 'send nothing', which the host would handle as requiring a re-send. I don't have a proper USB analyzer to verify exactly what is occurring.

Using the terminal emulators to generate data was really just a simulation of the 'Bootload application', which I found was terminating immediately after attempting to queue a 3.5KB transfer over the serial port. So eventually I have to move back to isolating the root cause of that.

Thanks for your help so far.
Best Regards.

[hathach]

@mws-rmain which board/mcu you are testing with, this info should always be presented in each issue/discussion. Each port will behave differently, but most user often forget this.

hardware endpoint will always NAK if not having any data queued up. There is zero chance tinyusb won't send NAK. You should look at other direction to troubleshoot this. If the issue occur with preempted rtos and/or multiple core mcu with fast enough data rate, there is a high chance it is a race. I just fixed an hard to find one with nrf52

• fix nrf easy dma race condition #1208

If you are certain it is a bug/race you could open an issue for it, make sure you add your own minimal example based on stock example (such as cdc_msc) to make sure we all testing the same firmware on the same hardware. Don't forget the steps/software to reproduce it.

[mws-rmain]

Thanks for the response, @hathach.
I did eventually resolve my issue as a bug in my UART FIFO routines, and I do have CDC data flowing. However, I still believe there is an issue, at least with the MCU I'm using ( SAMD21 - specifically SAMD21J18A & J17D). Initial development was on a SAMD21XPlained board, but now on a custom board.

My implementation is a CDC device, so for packet reception to an OUT endpoint the NAK should occur when a packet is received that the device cannot accept at the current time (i.e. the FIFO has no room for another packet). Since the FIFO is defined as 64 bytes, and the maximum packet size is 64 bytes, this will occur until the FIFO is empty.

During my investigations, I found that SAM D21 processor has a mechanism defined to NAK a packet when the device is not ready, which is NOT being used by the TinyUSB device driver. I could not find any mention of 'default NAK' or 'implicit NAK' behavior. Specifically, in the SAMD21 Family Datasheet (DS40001882G), section 32.6.2.7 'Management of OUT Transactions', states:

If EPSTATUS.STALLRQ0 in EPSTATUS is set, the incoming data is discarded. If the endpoint is not isochronous, 
a STALL handshake is returned to the host and the Transmit Stall Bank 0 interrupt bit in EPINTFLAG 
(EPINTFLAG.STALL0) is set.
For isochronous endpoints, data from both a DATA0 and DATA1 packet will be accepted. For other endpoint types
the PID is checked against EPSTATUS.DTGLOUT. If a PID mismatch occurs, the incoming data is discarded, and an 
ACK handshake is returned to the host.
**If EPSTATUS.BK0RDY is set, the incoming data is discarded, the bit Transmit Fail 0 interrupt bit in EPINTFLAG 
(EPINTFLAG.TRFAIL0) and the status bit STATUS_BK.ERRORFLOW are set. If the endpoint is not isochronous, a 
NAK handshake is returned to the host.**

TinyUSB does implement explicit STALL condition handling (usbd.c:1341, usbd_edpt_stall()) which manipulates the STALLRQ# bits in EPSTATUS register (dcd_samd.c:287, dcd_edpt_stall()), but I can't find any similar explicit handling of the NAK condition (which should be manipulating the BK#RDY bit in the EPSTATUS register).

TinyUSB does implement a 'busy' routine (usbd.c:1331, usbd_edpt_busy()), but this only sets internal flags - it does not call down to MCU specific functions - and it is not used at all by the CDC device.

I don't have a USB bus analyzer to verify it, but I believe TinyUSB CDC device will just fail to ACK the packet (i.e., the RED path in the image below, instead of the GREEN path), and the (windows USB) HOST driver may (by default) retry the packet anyway, and keeps doing so until all packets of a large block transfer are complete. If this is the case, the end result will be the same (all packets transferred), although there might be some side effects, such as unnecessary delays, etc.

Since I have something working, I'm moving ahead on my project. If you feel this is something that should be flagged for investigation, I can add it as an issue & document as suggested (please let me know).

[hathach]

I am glad you get it working. Your issue is probably in application code, you seem a bit confused with USB transfer which is totally normal. I got lots of confusion when starting with USB, it is a rather complicated protocol. Keep digging, I am sure you get it figured out sooner or later.

[mws-rmain]

I'd certainly like to correct course if I'm navigating down the wrong path!
Could you point out anything specific I've said that is incorrect with respect to handling of packet transfers to an OUT endpoint of a device?