UCT/IB/MLX5: Should rc_mlx5 support bounded batch CQ draining?

[Zacky-ran]

Background

In a high-completion-rate workload, we observed that calling progress several
times in one scheduling iteration improved throughput. The improvement became
smaller after a few progress calls.

This led us to look at the difference between the RC transports:

• rc_verbs can poll multiple work completions in one call through
  RX_MAX_POLL and TX_MAX_POLL.
• The rc_mlx5 Direct Verbs path normally processes one RX CQE and, when TX
  polling is selected, one TX CQE per interface progress call.

We would like to understand whether bounded batch CQ draining would be a
reasonable optimization for rc_mlx5.

Questions

1. Is processing one CQE per rc_mlx5 progress call an intentional part of
   the UCT design, for example to preserve latency or fairness?
2. Would it be consistent with the UCT progress model for rc_mlx5 to drain a
   bounded number of CQEs in one progress call?
3. If supported, should the limit reuse RX_MAX_POLL and TX_MAX_POLL, or
   should it be mlx5-specific?
4. Which layer would be preferred for this behavior: the mlx5 CQ polling
   helper, the rc_mlx5 RX/TX progress functions, or the upper worker
   progress loop?
5. Are there known constraints, such as RX/TX fairness, multiple interfaces on
   one worker, CQE zipping, MP-RQ or tag matching, that make batch draining
   unsuitable for rc_mlx5?