RFC: address "unknown TX stage" errors#372
Open
redlicha wants to merge 3 commits intov3io:developmentfrom
Open
RFC: address "unknown TX stage" errors#372redlicha wants to merge 3 commits intov3io:developmentfrom
redlicha wants to merge 3 commits intov3io:developmentfrom
Conversation
Signed-off-by: Arne Redlich <405850+redlicha@users.noreply.github.com>
Using list_first_entry_or_null on a list member will never return NULL, but rather either the next item if there is one or the head of the list if there's no item. Use list_next_entry_or_null instead. Signed-off-by: Arne Redlich <405850+redlicha@users.noreply.github.com>
`tx_ready_tasks_num` and `tx_ready_list` going out of sync after
network problems is the source of hitting "unknown TX stage" errors in
`xio_tcp_xmit`, which is caused by dereferencing an invalid `tcp_task`
pointer, since the list iteration is based on `tx_ready_tasks_num` and
not on the actual list:
```
(gdb) print tcp_hndl->tx_ready_tasks_num
$4 = 1
(gdb) print tcp_hndl->tx_ready_list
$5 = {next = 0x7fc02ecf2778, prev = 0x7fc02ecf2778}
```
Prior to [1] this would result in an infinite loop, with that commit
the connection is disconnected.
The exact reason for `tx_ready_tasks_num` and `tx_ready_list` going out
of sync is not clear, but code inspection suggests that in most places
we only need to know whether the `tx_ready_list` is empty or not, with
the exception of the following two annotated snippets in
`xio_tcp_xmit` that look at the actual value of `tx_ready_tasks_num`
while handling batching:
```
int xio_tcp_xmit(struct xio_tcp_transport *tcp_hndl)
{
while (likely(tcp_hndl->tx_ready_tasks_num &&
(tcp_hndl->tx_comp_cnt < COMPLETION_BATCH_MAX))) {
next_task = list_first_entry_or_null(&task->tasks_list_entry,
struct xio_task,
tasks_list_entry);
next_tcp_task = next_task ?
(struct xio_tcp_task *)next_task->dd_data : NULL;
// [...]
switch (tcp_task->txd.stage) {
// [...]
case XIO_TCP_TX_IN_SEND_CTL:
// [...]
++batch_count;
// - tx_ready_tasks_num > 0 (otherwise we'd
// not be in the loop anymore)
// - batch_count == tx_ready_tasks_num when
// next_task == NULL
// -> we can drop the batch_count != tx_ready_tasks_num
// check
if (batch_count != batch_nr &&
batch_count != tcp_hndl->tx_ready_tasks_num &&
next_task &&
(next_tcp_task->txd.stage ==
XIO_TCP_TX_IN_SEND_DATA) &&
(next_tcp_task->txd.msg.msg_iovlen +
tcp_hndl->tmp_work.msg_len) < IOV_MAX) {
task = next_task;
break;
}
// [...]
case XIO_TCP_TX_IN_SEND_DATA:
// [...]
// - tx_ready_tasks_num > 0 (otherwise we'd
// not be in the loop anymore)
// - batch_count == tx_ready_tasks_num when
// next_task == NULL
// -> we can drop the batch_count != tx_ready_tasks_num
// check
++batch_count;
if (batch_count != batch_nr &&
batch_count != tcp_hndl->tx_ready_tasks_num &&
next_task &&
(next_tcp_task->txd.stage ==
XIO_TCP_TX_IN_SEND_DATA) &&
(next_tcp_task->txd.msg.msg_iovlen +
tcp_hndl->tmp_work.msg_len) < IOV_MAX) {
task = next_task;
break;
}
// [...]
}
}
```
[1] "9d5f92077e645a6c7461dd4ab030df3a75417b2f accelio: disconnect on
tx stage error"
Signed-off-by: Arne Redlich <405850+redlicha@users.noreply.github.com>
Author
|
A polite "ping" in case this one slipped through the cracks :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims to address "unknown TX stage" errors observed in the TCP data path after network problems, which is not easily reproducible. Eventually inspection of such a case with gdb succeeded and showed that
tx_ready_tasks_numandtx_ready_listofstruct xio_tcp_transportwere out of sync. The reason for that isn't clear, but this PR takes the approach of preventing this altogether by removing the double bookkeeping and relying solely ontx_ready_listas the source of truth.It also emerged that there are uses of
list_first_entry_or_nullon a list member (instead of the list head) in order to find the next element in the list if any. This will never returnNULLbut instead misinterpret the list head.This hasn't seen much testing yet; I'm posting it as RFC to discuss if this strategy is valid.