core/api: Introduce fi_rpc

[j-xiong]

An RPC transaction consists of the client sending a request to the server, and the server sending back the response into the response buffer supplied by the client when the request was sent.

RPC operations can be built on top of existing point-to-point data transfer operations, including base messaging, tagged messaging, and RMA. One of the implementations uses base messaging for sending the request and tagged messaging with exact match for sending the response. One implication of such implementation is that the response messages should always arrive expected. In another word, if a reponse can not find a match it must be a stale message and can be safely dropped. However, currently the providers don't have a good way to obtain such information. As a result, those stale messages have to be kept indefinitely, hogging the resource, and negatively impacting the performance.

Having a dedicated RPC API allows the provider to understand the semanatics and perform optimizations when approciate.

The RPC API consist a set of calls to send RPC requests, a set of calls to send RPC response, a call to discard RPC requests, and a new CQ format to delivey RPC request meta data via completion entries. The content of the request and response are defined by the user.

Workflow of an RPC transacction:

Cleint:

• prepare request buffer
• fi_rpc

Server:

• fi_recv
• get completion (for fi_recv)
• check cqe.flags & FI_RPC
• get rpc_id and timeout from cqe
• process the received buffer ==> prepare response
• fi_rpc_resp
• get completion (for fi_rpc_resp)

Client:

• get completion (for fi_rpc)
• process the received response

A new primary capability FI_RPC is defined. It is also a flag used in completion entries indicating that the completed receive contains an RPC request.

[belynam]

@j-xiong Per discussion in today's OFIWG meeting, I have created #10966 to alter struct fi_opx_endpoint with a new rpc_reserved field that can be removed to allow for the larger size of struct fid_ep.

[shefty]

Need to examine RPC calls not having an fi_addr_t as input.