Why does the program directly call ucs_fatal to handle failures after ibv_post_send fails?

[super-train]

Hello everyone,

I have a question that has been bothering me for a long time. I would greatly appreciate any insights you can provide.

The specific issue is as follows: In version ucx-1.12, when using the RDMA send function uct_rc_verbs_ep_post_send, if ibv_post_send fails, the code directly calls ucs_fatal to handle it. This will terminate the program and throw a stack trace, which is unacceptable in practical applications. The corresponding code is as follows:

c
static UCS_F_ALWAYS_INLINE void
uct_rc_verbs_ep_post_send(uct_rc_verbs_iface_t* iface, uct_rc_verbs_ep_t* ep,
struct ibv_send_wr *wr, int send_flags, int max_log_sge)
{
struct ibv_send_wr *bad_wr;
int ret;

ucs_assertv(ep->qp->state == IBV_QPS_RTS, "QP 0x%x state is %d",
            ep->qp->qp_num, ep->qp->state);

if (!(send_flags & IBV_SEND_SIGNALED)) {
    send_flags |= uct_rc_iface_tx_moderation(&iface->super, &ep->super.txqp,
                                             IBV_SEND_SIGNALED);
}
if (wr->opcode == IBV_WR_RDMA_READ) {
    send_flags |= uct_rc_ep_fm(&iface->super, &ep->fi, IBV_SEND_FENCE);
}

wr->send_flags = send_flags;
wr->wr_id      = ep->txcnt.pi + 1;

uct_ib_log_post_send(&iface->super.super, ep->qp, wr, max_log_sge,
                     (wr->opcode == IBV_WR_SEND) ? uct_rc_ep_packet_dump : NULL);

ret = ibv_post_send(ep->qp, wr, &bad_wr);
if (ret != 0) {
    ucs_fatal("ibv_post_send() returned %d (%m)", ret);
}

uct_rc_verbs_txqp_posted(&ep->super.txqp, &ep->txcnt, &iface->super, send_flags & IBV_SEND_SIGNALED);

}
My questions are as follows:

Why does it directly call ucs_fatal here? This is very ungraceful and unacceptable in practical implementation.
If I want to create a new function similar to uct_rc_verbs_iface_poll_tx that handles failures when ibv_post_send fails, similar to how it invokes iface->super.super.ops->handle_failure(&iface->super.super, &wc[i], status);, would that be feasible? If so, what resources should be handled within this new post-send handling function?
c
static UCS_F_ALWAYS_INLINE unsigned
uct_rc_verbs_iface_poll_tx(uct_rc_verbs_iface_t *iface)
{
uct_rc_verbs_ep_t *ep;
uint16_t count;
int i;
unsigned num_wcs = iface->super.super.config.tx_max_poll;
struct ibv_wc wc[num_wcs];
ucs_status_t status;

UCT_RC_VERBS_IFACE_FOREACH_TXWQE(&iface->super, i, wc, num_wcs) {
    ep = ucs_derived_of(uct_rc_iface_lookup_ep(&iface->super, wc[i].qp_num),
                        uct_rc_verbs_ep_t);
    if (ucs_unlikely((wc[i].status != IBV_WC_SUCCESS) || (ep == NULL))) {
        status = uct_rc_verbs_wc_to_ucs_status(wc[i].status);
        iface->super.super.ops->handle_failure(&iface->super.super, &wc[i],
                                               status);
        continue;
    }

    count = uct_rc_verbs_get_tx_res_count(ep, &wc[i]);
    ucs_trace_poll("rc_verbs iface %p tx_wc wrid 0x%lx ep %p qpn 0x%x count %d",
                   iface, wc[i].wr_id, ep, wc[i].qp_num, count);

    uct_rc_txqp_completion_desc(&ep->super.txqp, ep->txcnt.ci + count);
    ucs_arbiter_group_schedule(&iface->super.tx.arbiter,
                               &ep->super.arb_group);
    uct_rc_verbs_update_tx_res(&iface->super, ep, count);
    ucs_arbiter_dispatch(&iface->super.tx.arbiter, 1, uct_rc_ep_process_pending,
                         NULL);
}

return num_wcs;

}
Additionally, I have checked the source code in ucx-1.18.0, and it also directly calls ucs_fatal when ibv_post_send fails. Why is it that the implementation is consistently terminating the program like this? Whereas in SPDK's NVMe and 3FS, the handling of ibv_post_send failures involves changing the QP state to unavailable and releasing resources. I'm puzzled about why UCX handles it differently. If you have any insights, please let me know. Thank you!