STM32 usb networking stack threading issue

[Coi-l]

Describe the bug
I am using RNDIS/ECM on an stm32 nucleo-h743zi2 board.
I am using civetweb as a webserver.
I have a html page which in turn loads a number of other javascript files.
When I am using only 1 worker thread in civetweb everything works perfectly, but a bit slow due to only one thread.
When I try to increase this to 2 or more threads I get decoding/loading issues on the files that are loaded after the first html file is loaded. The files are corrupt and can't be completely unpacked by the browser.

When I try the same but using the RJ45 connection and normal ethernet driver on the same board it works perfectly.
I can make the USB connection work by adding a lot of debug logs while sending the data.

I've tested the USB connection on a nrf board where it works without issue.

This leads me to believe it is a problem with the thread safety of the usb networking stack on the stm32 platform.

To Reproduce
See above

Expected behavior
Pages should be sent without error

Impact
showstopper

Logs and console output

Environment (please complete the following information):

• OS: Linux
• Toolchain: Zephyr SDK
• zephyr-v3.2.0

Additional context

[nordicjm]

I am using civetweb as a webserver.
zephyr-v3.2.0

These two statements are at odds with each other, civetweb was unmaintained for a few releases and was removed prior to the 3.2 release. Please fix your post

[erwango]

@Coi-l Would you be able to share a project that could be used to reproduce the issue you're seeing?

[Coi-l]

I am using civetweb as a webserver.
zephyr-v3.2.0

These two statements are at odds with each other, civetweb was unmaintained for a few releases and was removed prior to the 3.2 release. Please fix your post

I'm not sure why you have such an aggressive tone.
While civetweb was indeed deprecated and removed between 3.1 and 3.2, it does not mean that it is impossible to make civetweb work with zephyr. It took less than an hour to get civetweb working again after I upgraded to 3.2.

While I am new to zephyr community interactions I sincerely hope that your attitude is not representative for the rest of the community here.

[Coi-l]

@Coi-l Would you be able to share a project that could be used to reproduce the issue you're seeing?

I can't share the full project. I will try to put together a smaller example.

[jfischer-no]

I'm not sure why you have such an aggressive tone.
While civetweb was indeed deprecated and removed between 3.1 and 3.2, it does not mean that it is impossible to make civetweb work with zephyr. It took less than an hour to get civetweb working again after I upgraded to 3.2.

However, you should understand that you are probably reporting a bug in your project to upstream. There is no more support for civetweb, your description is very vague, and claims to be a bug in upstream. Please report it so that it can be reproduced with few changes in the tree, without external dependencies (like civetweb).

@erwango +1 to move to discussion.

[Coi-l]

However, you should understand that you are probably reporting a bug in your project to upstream. There is no more support for civetweb, your description is very vague, and claims to be a bug in upstream. Please report it so that it can be reproduced with few changes in the tree, without external dependencies (like civetweb).

@erwango +1 to move to discussion.

I understand that. I am very open for the possibility that it is a bug in my own project/civetweb integration. However the behaviour I saw at least in my opinion hinted at a bug somewhere in the stm32 networked usb stack.
If I should have filed this not under bug but somewhere else I apologize. I am fairly new to using github for partaking in a community. I can see now that there is a Discussions tab, please move it there if that is better suited for my question.

[erwango]

ok, moving to discussion until a genuine upstream bug is characterized

[Coi-l]

I've done some more poking around and have a branch with the problem here:
https://github.com/skunkcollective/zephyr/tree/usb-web-problem
The example is a modified civetweb/websocket example where I've extended it with more files to load from the index.html page.

Any changes to the prj.conf is at the end.
I've tested this on linux/firefox/chrome.
The branch requires a change in civetweb.c to work (see commit message). It is due to the stack size of the civetweb threads.
You need to manually set your ip address on the new network interface. The board is running on 192.0.2.1/24. Port 8080.
You can bring up the developer console in firefox to see the file loading if you are familiar with that.
For ease of use this branch is base on 3.1.0 but I see the same behavior on 3.2.0

I am no longer sure that it is a threading issue, but might be related to memory instead. Or more likely something I don't understand regarding the zephyr stack.

Running the example as it is will caues a lot of errors and warnings in the usb stack:
[00:00:49.865,000] usb_rndis: Not enough pkt buffers, len 471, skip 407
[00:00:49.865,000] usb_rndis: Skip 64 bytes out of remaining 407 bytes
[00:00:49.865,000] usb_rndis: Skip 64 bytes out of remaining 343 bytes
[00:00:49.866,000] usb_rndis: Skip 64 bytes out of remaining 279 bytes
[00:00:49.866,000] usb_rndis: Skip 64 bytes out of remaining 215 bytes
[00:00:49.866,000] usb_rndis: Skip 64 bytes out of remaining 151 bytes
[00:00:49.866,000] usb_rndis: Skip 64 bytes out of remaining 87 bytes
[00:00:49.866,000] usb_rndis: Skip 23 bytes out of remaining 23 bytes

This seems to be related to receiving, which is a bit weird since there is not much receiving going on.

and also this:
[00:00:50.586,000] http_server_handlers: aborted! ret: -1
This is a civetweb error produced when it received error from send().

So, there seems to be too little memory. My naive assumption was that the network/tcp stack would chunk this up in proper chunks and feed the usb drivers in proper chunks as well.

The problems seems to go away with increasing the memory with these configs:

CONFIG_NET_PKT_RX_COUNT=25
CONFIG_NET_BUF_RX_COUNT=72
CONFIG_NET_PKT_TX_COUNT=25
CONFIG_NET_BUF_TX_COUNT=72

available in the prj.conf file.

However, I don't really understand why these would need to be increased, nor how I should calculate what they need to be.

Any help or insight into this would be appreciated.
Please also remember that I did not have to increase these values in the nrf usb stack nor if I was running on the normal ethernet.

[Coi-l]

In there, you have a 287KB javascript file... I think you're vastly overthinking the abilities or underestimating the requirements for files like that on a small cortex processor.

Hm?
The javascript file is there to have something to send. And it was the easiest to make sure the browser would load.
The javascript is also executed on the browser that loads it not putting any load on the board. The only load is just to send the file.

[jfischer-no]

[00:00:49.865,000] usb_rndis: Not enough pkt buffers, len 471, skip 407 coming from

zephyr/subsys/usb/device/class/netusb/function_rndis.c

Lines 382 to 393 in 5d9c65f

	pkt = net_pkt_rx_alloc_with_buffer(netusb_net_iface(),
	rndis.in_pkt_len, AF_UNSPEC,
	0, K_NO_WAIT);
	if (!pkt) {
	/* In case of low memory: skip the whole packet
	* hoping to get buffers for later ones
	*/
	rndis.skip_bytes = rndis.in_pkt_len - len;
	rndis.rx_no_buf++;
	LOG_ERR("Not enough pkt buffers, len %u, skip %u",
	rndis.in_pkt_len, rndis.skip_bytes);

means there are not enough pkts provided by the network layer. You can try to increase CONFIG_NET_PKT_RX_COUNT and CONFIG_NET_BUF_RX_COUNT.

[Coi-l]

@jfischer-no Yes. As I wrote in the post it does go away when I increase the available packet buffers. But is this really the correct way to go about it? Just increase the memory until it works? It feels a bit strange to me. Is there no flow control that can be applied or something similar?

[jfischer-no]

But is this really the correct way to go about it? Just increase the memory until it works?

Yes,

It feels a bit strange to me. Is there no flow control that can be applied or something similar?

This RNDIS implementation could be reworked to offload net_pkt allocation in a thread or work and block/reschedule-work there until net_pkt is available. It also needs to be synchronized in a way that controller NACKs OUT bulk transactions. I do not think it make sense to spend resources and time on it, RNDIS implementation will likely be replaced by CDC NCM in new USB support.

[Coi-l]

Ok, thank you @jfischer-no