RTU Master cannot keep up with (very) responsive slave or bus spikes/faulty bytes (EPROT-16)

[Wisje]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

often

Expected behavior

Normal operation, transmit request and receive fast response.
When a faulty byte is detected, ignore and continue.

Actual behavior (suspected bug)

Two things happen:

1. When the slave response is too fast, the semaphore in not released in mb_port_ser_send_data before a frame is received. mb_port_ser_task receives UART_DATA event, semaphore is not free, data is flushed.
2. On a bus spike/faulty byte received, UART_DATA event drops the short package, but also flushes the fifo and uart queue which could hold the correct data.

Both result in ESP_ERR_TIMEOUT because data is flushed.

Error logs or terminal output

I (1612982) mb_port.serial: mbm_rtu@0x3ffd454c, fragmented packet 13 byte(s)
E (1613982) MB_CONTROLLER_MASTER: mbc_master_get_parameter(131): Master get parameter failure, error=(0x107) (ESP_ERR_TIMEOUT).
E (1613982) MODBUS: Characteristic #10 MainRead (read), read parameter read fail.

E (92902) MB_CONTROLLER_MASTER: mbc_master_get_parameter(131): Master get parameter failure, error=(0x107) (ESP_ERR_TIMEOUT).
E (92902) MODBUS: Characteristic #10 MainRead (read), read parameter read fail.

Steps to reproduce the behavior

Two devices with bus-driver with DE & RE connected, so no echo on Rx.
Slave should respond directly when request is received.

Faulty byte happens sometimes in this setup as well, always a 0x00 byte. But could be introduced externally on the bus as well.

Project release version

v2.2.1

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Windows

Operating system version

Windows 11

Shell

PowerShell

Additional context

The faulty 0x00 byte can be suppressed when setting bit UART_RS485TX_RX_EN. Byte could from the switching? 0x00 faulty byte is not seen on the bus, just received internal.

Proposed modification for mb_port_ser_task:

• Do not flush. Just read the data length from the current event to throw away and break. This means other queued events are handled accordingly.
• Option to remove mb_port_ser_bus_sema_is_busy check on systems without echo.

The slave device sometime is even faster than the RTS release. Looking to change to using DTR as api says this is done by hardware. I think this is outside the modbus. A topic is started on the esp forum (ESP32 RS485 DTR Pin).

[alisitsyn]

@Wisje ,

The faulty 0x00 byte can be suppressed when setting bit UART_RS485TX_RX_EN. Byte could from the switching? 0x00 faulty byte is not seen on the bus, just received internal.

Thank you for the issue. I am aware of this situation. The 0x00 byte comes sometime due to bus transition from TX to RX. This is related to several factors including bus connection, driver type, schematic fail-safe biasing and software aspects of UART. Unfortunately, I can not give you universal recommendation to reliably avoid this.

In order to avoid known user issues the Modbus port uses the approach implemented on the port layer. If during initial timeout two continuous T3.5 timeout events are detected by the driver, it will flush the buffer and treat this as a fragmentation of data or race condition. The small frames with T3.5 event are also flushed. This is not an ideal solution but allows avoiding known issues with reception of Modbus frames and avoid race conditions. It is decided that it is safer to flush (skip) one request in the buffer, do retry than try to repair the frame and get the risk of possible race conditions.

Do not flush. Just read the data length from the current event to throw away and break. This means other queued events are handled accordingly. Option to remove mb_port_ser_bus_sema_is_busy check on systems without echo.

This functionality was implemented to address the user issues observed during long time. I think it is possible to improve this functionality adding the option and make a fix of this behavior for "responsive" slaves but it needs to cover all aspects of use from user side. The changes need to cover many specific aspects of user environment, software, race conditions and may take additional comprehensive testing. This will be planned accordingly.

The slave device sometime is even faster than the RTS release. Looking to change to using DTR as api says this is done by hardware. I think this is outside the modbus. A topic is started on the esp forum (ESP32 RS485 DTR Pin).

I would agree with your chose to use the DTR pin to switch the direction. It is controlled by hardware and would allow to avoid some issues for responsive systems since the RTS is controlled by software of driver. The steps to follow:

1. Check the initialization of UART in modbus example as below:

    // Set UART pin numbers
    // err = uart_set_pin(MB_PORT_NUM, CONFIG_MB_UART_TXD, CONFIG_MB_UART_RXD,
    //                           CONFIG_MB_UART_RTS, UART_PIN_NO_CHANGE); <<<<<<<< comment out the regular 4 pin function;
    // <<<<<<<<<<<<<< use the 6 pin based setup for UART as below:
    err = uart_set_pin(MB_PORT_NUM, CONFIG_MB_UART_TXD, CONFIG_MB_UART_RXD, 
                              CONFIG_MB_UART_RTS, UART_PIN_NO_CHANGE, MB_UART_DTR, UART_PIN_NO_CHANGE);

    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE, TAG,
                       "mb serial set pin failure, uart_set_pin() returned (0x%x).", (int)err);

    err = mbc_master_start(master_handle);
    MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE, TAG,
                       "mb controller start fail, returned (0x%x).", (int)err);

    // Set driver mode to Half Duplex
    //err = uart_set_mode(MB_PORT_NUM, UART_MODE_RS485_HALF_DUPLEX); // <<<<< comment out the setting of HALF_DUPLEX mode of UART.
    //MB_RETURN_ON_FALSE((err == ESP_OK), ESP_ERR_INVALID_STATE, TAG,
    //                   "mb serial set mode failure, uart_set_mode() returned (0x%x).", (int)err);  
    // The DTR pin will be controlled by hardware on MB_UART_DTR pin of ESP32 and needs to be connected to DE pin of line driver instead of RTS pin

2. Note: The hardware control over DTR pin on ESP32 can be used only on UART0. So, you need to select the Modbus port MB_PORT_NUM=0, and change the default console settings to use the UART1 instead of UART0.

Let me know if you have questions or other comments.
Thanks.

[Wisje]

Thanks for the fast and clear response.
I understand the flush and the safer option choices, I think for true fragmented packages, this is the best option.

My problem is mostly with the 0x00 byte. While I would like to eliminate the byte altogether, I think software could flush/ignore this by modifying the flush function.
The 0x00 byte generates a T3.5 event and is treated as a fragmented package and flushed, which is good. But while flushing is queued, the UART can be receiving the response. And during the flush, which could happen later due to tasking, the complete UART buffer is flushed which could hold bytes received of the (not yet completed) response. I think if it only flushes the data received until the T3.5 event, you can only throw away the faulty byte, and continue operating. No need to repair the package, because that is still received in one T3.5 event.

I would agree with your chose to use the DTR pin to switch the direction. It is controlled by hardware and would allow to avoid some issues for responsive systems since the RTS is controlled by software of driver. The steps to follow:

Is there a different driver I need for the 6 pin setup? I've found the function in the API, but it does not compile or show up in the driver. Only mention of the DTR pin is uart_set_dtr.

Thanks

[alisitsyn]

My problem is mostly with the 0x00 byte. While I would like to eliminate the byte altogether, I think software could flush/ignore this by modifying the flush function.

The 0x00 byte is received due to complex criteria (could be a short spike on RX due to rx/tx transition + UART controller + software processing and etc.) and need to be investigated properly. This byte is not properly received by UART and as I remember it generates the RX BRK_DET interrupt but then byte is saved into FIFO buffer in spite of error and being queued into ring buffer. I think the byte can be dropped by UART controller with its configuration bits (will check some idea). I am not sure that this parasitic byte is generated with separate T3.5 event and can be easily flushed. It is supposed that the T3.5 event is generated later when the whole buffer is received.

According to above, I think it is better to filter the byte on the driver layer. The flushing of parasitic frame or even filtering can be implemented on port layer but this is not good solution. I confirm this known issue which had to be postponed for long time.
The related investigation will be renewed as soon as possible. I would appreciate it if you are able to share any additional investigation results (oscilloscope pictures or logs ) according to it.

Is there a different driver I need for the 6 pin setup? I've found the function in the API, but it does not compile or show up in the driver. Only mention of the DTR pin is uart_set_dtr.

The driver was updated recently to address the direction control over DTR pin (updated as per esp-idf v6.0 update). In the new driver the uart_set_pin API is now a macro that can receive 4 pin configuration or 6 pin configuration and calls appropriate function. So, you need just to provide correct number of pins and do not need to call the 6pin function which is in private scope.

Please refer to:
UART RS485 half-duplex intermittent DE line
UART driver update

Thanks.

[tanglebridge]

I think i have the same problem. I am on version 2.1.1 of esp-modbus. I added some log messages and see the data arrive before the semaphore is released.

The board is a T-CAN485 (esp32). It is communicating with an Epever solar charger (TracerBP). I am attaching a capture from the oscilloscope. A and B are yellow and blue, magenta is the RTS signal. The bottom panel shows the difference between the signal channels. You see the signal being sent and the reply. There are 2.5 milliseconds between the end of the request and the start of the reply. Baudrate is 115200/seconds, so 3.5T is about 0.3 milliseconds, so that should not be an issue.

I verified the answer bytes and they are correct. Plain UART based communication works.

In mb_port_ser_task() if I comment out lines 159 and 160:

                            mb_port_ser_rx_flush(&port_obj->base);
                            break;

the message is processed and interpreted without issue.

[alisitsyn]

@tanglebridge,

Thank you for the clear description. I can agree that the better flushing algorithm should be implemented to allow the response from the slave to be registered. From the component standpoint, it can not be just removing the mb_port_ser_rx_flush(&port_obj->base); because this will return the other user issues. Also, the driver needs to be updated to remove the zero byte from the bus, which applies to the ESP32 target only. This will take time because I am very busy. Could removing flushing be a temporary local solution for your project while the issue is not addressed?

[tanglebridge]

Yes - that would be a solution, and I will probably do that. Thank you.

It seems to me the decision to flush would be better handled after a failed attempt to interpret the byte stream. After all the CRC at the end will indicate whether the message is intact or not. This basically means suspending that decision here and flushing at the interpretation level. With some flexibility that could even handle a zero byte at the end.

Could you point me to the code that interprets the bytestream and checks CRC? I suppose this approach would work for RTU but not for ASCII?

[alisitsyn]

Could you point me to the code that interprets the bytestream and checks CRC? I suppose this approach would work for RTU but not for ASCII?

Yes, the zero byte will be dropped on RTU transport layer but the request will fail and will need to retry. It always first byte in the response frame and can be effectively filtered on UART layer or port layer.

https://github.com/espressif/esp-modbus/blob/main/modbus/mb_transports/rtu/rtu_slave.c#L148
https://github.com/espressif/esp-modbus/blob/main/modbus/mb_transports/rtu/rtu_master.c#L163

The flushing is still needed to avoid the race condition when the expired response from slave for previous transaction comes right after start of next request. The modbus serial ADU does not have the TID and if the next packet has the same command, offset, length fields it can be treated by stack as a response to current request. The flushing algorithm can drop expired response and also allows to drop fragmented responses during transaction life time. The additional mechanism is implemented with injection of unique TID as a timestamp into FSM for better control of transaction life time.

[tanglebridge]

I understand.

In the current design the processing of the RX buffer by the modbus queue handler is simply too slow. I see two options:

1. Trigger mb_port_ser_task() each time the RX TOUT is reached;
2. Separate UART queue operations from modbus queue operations.

(1) would simply minimize the processing delay and probably dramatically improve throughput as well as latency, but I don't know if the code architecture supports this change. This approach has some other advantages (smaller queues for one).

For (2): The modbus frame does contain the ID of the slave that is replying, but nothing else. I would argue that the master could enforce clean queuing of message-response pairs, meaning that the next queued message is sent only when the previous has received its reply (or times out). Maybe this could be an option that is configured for each slave ID. This inverts what you are doing now. Instead of allowing out-off-sync replies that in some cases break communication completely and otherwise reduce overall performance, you take an average performance hit on latency (but not throughput).

To me, the (serial) modbus protocol basically assumes (2) is done. I am not familiar with the TPC/IP based one, but by making queuing enforcement an option you can serve both scenarios. Even if (1) is implemented - seems like a good idea in general - then (2) should still be considered since the modbus protocol more or less implies this as a requirement.

[alisitsyn]

For (2): The modbus frame does contain the ID of the slave that is replying, but nothing else.

• The frame contains UID which is not a unique identifier of the frame. What I meant is that it does not help to distinguish the correct response in case of a race condition.

I would argue that the master could enforce clean queuing of message-response pairs, meaning that the next queued message is sent only when the previous has received its reply (or times out).

This approach is widely used, but in the case of a serial port, an additional queue is useless for queuing of request-response pairs because the lifetime of the request buffer is limited by the time of transaction, but the response data is queued by the UART driver ring buffer. This approach is selected to decrease memory allocation or fragmentation.

I would argue that the master could enforce clean queuing of message-response pairs, meaning that the next queued message is sent only when the previous has received its reply (or times out).

The current implementation waits for the response during the configured timeout, which is determined by the timer. So, this corresponds to the specification but does not allow avoiding race conditions, which can be resolved only on the user app layer, which I do not control.

meaning that the next queued message is sent only when the previous has received its reply (or times out). Maybe this could be an option that is configured for each slave ID. This inverts what you are doing now. Instead of allowing out-off-sync replies that in some cases break communication completely and otherwise reduce overall performance, you take an average performance hit on latency (but not throughput).

If I understand you correctly, this will not satisfy the specification, meaning it is unusable for the stack.

To me, the (serial) modbus protocol basically assumes (2) is done. I am not familiar with the TPC/IP based one, but by making queuing enforcement an option you can serve both scenarios.

The TCP queues data as paired request-response frames, usually used for serial as well. However, it was the design decision to not use a separate queue for the above reason.

Even if (1) is implemented—it seems like a good idea in general - then (2) should still be considered since the modbus protocol more or less implies this as a requirement.

1. Trigger mb_port_ser_task() each time the RX TOUT is reached;

In the current design the processing of the RX buffer by the modbus queue handler is simply too slow.

I think I did not catch your suggestion. On the T3.5 timeout event, the UART event queue unblocks the mb_port_ser_task() in the current implementation to process the received buffer that corresponds to the event. So, the task is already triggered on each RX_TOUT event.

I will try to verify and improve the slow processing of the Modbus object layer. The latency is mostly related to consequent processing of FSM events, the 2) can be considered but will not solve this. Other algorithms are possible but this stack is designed to use RTOS based approach and polling algorithm without using interrupts or high prio tasks to allow portability. However, other esp components or user app tasks can use high prio and interrupts. When the event is ready a higher-priority task (e.g., WiFi, Bluetooth, or app_main) may be running and scheduler lets the higher-priority task to finish its work before return to modbus task. This is where the few milliseconds latency is added into processing.

Thank you for your comments. I will take it into account during the update.

[alisitsyn]

@tanglebridge,

I was in a hurry yesterday and did not catch the whole idea with 'inverted logic' + 1) and 2). I think I understand the approach with queuing the pair of request - response when the response is received on RX_TOUT (which corresponds to the configured slave response timeout) event. It could allow getting more control for received messages and can even help to drop the fragmented or expired response (if exists) but take into account the latest one.

The approach has several pros and cons (significant is throughput) and needs to be verified.

In the current design the processing of the RX buffer by the modbus queue handler is simply too slow.

I am not sure if the new approach would allow to increase the handling significantly.

[tanglebridge]

I was tired and used too many words - sorry. Basically, the MODBUS protocol assumes one master on each bus, and each slave only replies when it gets a request from the master. The method mbc_master_get_parameter() only returns once the answer is provided, so strictly request-response for each slave. On my ESP32 that takes over 100 milliseconds. On the oscilloscope the answer appears within 2.5 milliseconds, which is well over 3.5T at 115200 baud. The UART correctly stores the response as it arrives.

This means:

1. the code is too slow in processing the UART response;
2. this slowness in the current architecture causes the correct response in the UART to be interpreted as bus interference.

To fix this:

1. move the UART response from UART to MODBUS memory immediately on reception (3.5T timeout);
2. you can use a queue on the modbus side to buffer transferred (UART) messages to process;
3. simpler: don't permit new transmission until the last message was processed (no queue).

You can delay the processing of the MODBUS messages, but you can't delay moving it from UART to the MODBUS buffer. I also think that you cannot reasonably detect bus interference where it is done now. Instead I suggest one of the following to indicate bus interferece:

1. out-of-sequence response: slave X replied to a request for slave Y;
2. MODBUS message corruption (CRC error).

Both of these may be due to bus interference, but I don't think you can decide that before actually looking at the bytes.

I hope this is a bit more clear.

[tanglebridge]

Incidentally: with the flushing code commented out the system is working flawlessly, but that is with a single slave device.