RDMA編程基礎(chǔ)
存儲(chǔ)大師班 | RDMA簡(jiǎn)介與編程基礎(chǔ) -https://zhuanlan.zhihu.com/p/387549948

1. RDMA的學(xué)習(xí)環(huán)境搭建
RDMA需要專門(mén)的RDMA網(wǎng)卡或者InfiniBand卡才能使用，學(xué)習(xí)RDMA而又沒(méi)有這些硬件設(shè)備，可以使用一個(gè)軟件RDMA模擬環(huán)境，softiwarp ，
- 這是加載地址：https://github.com/zrlio/softiwarp
- 這是安裝教程：http://www.reflectionsofthevoid.com/2011/03/how-to-install-soft-iwarp-on-ubuntu.html

更多的rdmacm實(shí)例：，
- https://github.com/tarickb/the-geek-in-the-corner
需要注意的是，這個(gè)例子里面缺省用的是IPv6連接，如果希望在IPv4環(huán)境下測(cè)試，需要先改代碼用IPv4地址。

2. RDMA與socket的類比
和Socket連接類似，RDMA連接也分為可靠連接和不可靠連接。然而也不完全相同，Socket的可靠連接就是TCP連接，是流式的；不可靠連接也就是UDP，是消息式的。對(duì)于RDMA來(lái)說(shuō)，無(wú)論是可靠連接和不可靠連接，都是消息式的。
編程角度看，RDMA代碼也分為Server端，Client端，也有bind, listen, connect, accept,等動(dòng)作，然而細(xì)節(jié)上仍有不少區(qū)別。

大家可以關(guān)注一下mellonx的vma，貌似可以直接用socket api通信，方便很多：
【RDMA】降低CPU除了RDMA (vbers）還是VMA ？|使用socket進(jìn)行RDMA編程？_bandaoyu的note-CSDN博客
前言看介紹，像是mellonx針對(duì)其kernel bypass網(wǎng)卡（RDMA網(wǎng)卡）提供的一個(gè)lib庫(kù)，該lib庫(kù)對(duì)外提供socket api，使得用戶的程序不需要修改就可以直接使用kernel bypass網(wǎng)卡（如RDMA網(wǎng)卡）。我們都知道RDMA 網(wǎng)卡目前使用的是rdma_cm和vbers api編程，和socket不一樣，如果能用socket對(duì)RDMA編程，那確實(shí)是很大的利好。官網(wǎng)介紹什么是VMA？Mellanox Interconnect Community官方介紹：M
https://blog.csdn.net/bandaoyu/article/details/120726746

rdma_cm API說(shuō)明：

https://linux.die.net/man/3/rdma_create_id （推薦）

https://www.ibm.com/docs/en/aix/7.2?topic=operations-rdma-listen （內(nèi)容少）

rdma_cm API 管理連接（建立連接和銷毀）+vbers api 管理收發(fā)

RDMA主機(jī)使用queue pairs（QP）進(jìn)行通信；主機(jī)創(chuàng)建由發(fā)送隊(duì)列SQ和接收隊(duì)列RQ組成的QP，并使用verbs API將操作post 到這些隊(duì)列。（所以rdma_cm是管理連接的，收發(fā)還是verbs API。）

3. RDMA服務(wù)器的代碼流程
main（）
{
channel=rdma_create_event_channel
這一步是創(chuàng)建一個(gè)event channel，event channel是RDMA設(shè)備在操作完成后，或者有連接請(qǐng)求等事件發(fā)生時(shí)，用來(lái)通知應(yīng)用程序的通道。其內(nèi)部就是一個(gè)file descriptor, 因此可以進(jìn)行poll等操作。

rdma_create_id(channel, **id，……)
這一步創(chuàng)建一個(gè)rdma_cm_id, 概念上等價(jià)與socket編程時(shí)的listen socket。

rdma_bind_addr(id,addr)
和socket編程一樣，也要先綁定一個(gè)本地的地址和端口，以進(jìn)行l(wèi)isten操作。

rdma_listen(id,block)
開(kāi)始偵聽(tīng)客戶端的連接請(qǐng)求

rdma_get_cm_event（channel，&event）
這個(gè)調(diào)用就是作用在第一步創(chuàng)建的event channel上面，要從event channel中獲取一個(gè)事件。這是個(gè)阻塞調(diào)用，只有有事件時(shí)才會(huì)返回。在一切正常的情況下，函數(shù)返回時(shí)會(huì)得到一個(gè) RDMA_CM_EVENT_CONNECT_REQUEST事件，也就是說(shuō)，有客戶端發(fā)起連接了。
在事件的參數(shù)里面，會(huì)有一個(gè)新的rdma_cm_id傳入。這點(diǎn)和socket是不同的，socket只有在accept后才有新的socket fd創(chuàng)建。

on_event()

{

? ? ?on_connect_request()//RDMA_CM_EVENT_CONNECT_REQUEST

? ? {

? ? ? ? build_context()

? ? ? {

6.ibv_alloc_pd

創(chuàng)建一個(gè)protection domain。protection domain可以看作是一個(gè)內(nèi)存保護(hù)單位，在內(nèi)存區(qū)域和隊(duì)列直接建立一個(gè)關(guān)聯(lián)關(guān)系，防止未授權(quán)的訪問(wèn)。

7.ibv_create_comp_channel

和之前創(chuàng)建的event channel類似，這也是一個(gè)event channel，但只用來(lái)報(bào)告【完成隊(duì)列】里面的事件。當(dāng)【完成隊(duì)列】里有新的任務(wù)完成時(shí)，就通過(guò)這個(gè)channel向應(yīng)用程序報(bào)告。

8.ibv_create_cq

創(chuàng)建【完成隊(duì)列】，創(chuàng)建時(shí)就指定使用第6步的channel。

}//--end build_context()

9.rdma_create_qp
創(chuàng)建一個(gè)queue pair, 一個(gè)queue pair包括一個(gè)發(fā)送queue和一個(gè)接收queue. 指定使用前面創(chuàng)建的cq作為完成隊(duì)列。該qp創(chuàng)建時(shí)就指定關(guān)聯(lián)到第6步創(chuàng)建的pd上。

10.ibv_reg_mr
注冊(cè)內(nèi)存區(qū)域。RDMA使用的內(nèi)存，必須事先進(jìn)行注冊(cè)。這個(gè)是可以理解的，DMA的內(nèi)存在邊界對(duì)齊，能否被swap等方面，都有要求。

11.rdma_accept
至此，做好了全部的準(zhǔn)備工作，可以調(diào)用accept接受客戶端的這個(gè)請(qǐng)求了。 –：）長(zhǎng)出一口氣 ~~ 且慢，

}

//--end on_connect_request()

12.rdma_ack_cm_event
對(duì)于每個(gè)從event channel得到的事件，都要調(diào)用ack函數(shù)，否則會(huì)產(chǎn)生內(nèi)存泄漏。這一步的ack是對(duì)應(yīng)第5步的get。每一次get調(diào)用，都要有對(duì)應(yīng)的ack調(diào)用。

13.rdma_get_cm_event
繼續(xù)調(diào)用rdma_get_cm_event, 一切正常的話我們此時(shí)應(yīng)該得到 RDMA_CM_EVENT_ESTABLISHED 事件，表示連接已經(jīng)建立起來(lái)。不需要做額外的處理，直接rdma_ack_cm_event就行了

}//--end on_event()

終于可以開(kāi)始進(jìn)行數(shù)據(jù)傳輸了 ==== (如何傳輸下篇再說(shuō))

參考：http://10.165.104.246:8080/#/c/43882/

4. 關(guān)閉連接

斷開(kāi)連接
當(dāng)rdma_get_cm_event返回RDMA_CM_EVENT_DISCONNECTED事件時(shí)，表示客戶端斷開(kāi)了連接，server端要進(jìn)行對(duì)應(yīng)的清理。此時(shí)可以調(diào)用rdma_ack_cm_event釋放事件資源。然后依次調(diào)用下面的函數(shù)，釋放連接資源，內(nèi)存資源，隊(duì)列資源。

rdma_disconnect

rdma_destroy_qp

ibv_dereg_mr

rdma_destroy_id
釋放同客戶端連接的rdma_cm_id

rdma_destroy_id
釋放用于偵聽(tīng)的rdma_cm_id

rdma_destroy_event_channel
釋放 event channel

}
// end main

實(shí)例
源碼地址- https://github.com/tarickb/the-geek-in-the-corner

用法
[root@localhost 01_basic-client-server]# ./server?
listening on port 42956.

client <server-address> <server-port>
?

Makefile
.PHONY: clean

CFLAGS ?:= -Wall -g
LDLIBS ?:= ${LDLIBS} -lrdmacm -libverbs -lpthread

APPS ? ?:= server client

all: ${APPS}

clean:
? ? rm -f ${APPS}

注意：makefile 沒(méi)有-L 指定lib的路徑，所以 -lrdmacm -libverbs -lpthread 對(duì)應(yīng)的庫(kù) librdmacm.so libibverbs.so libpthread.so 應(yīng)放在默認(rèn)的路徑下/usr/lib 或/usr/lib64

服務(wù)端server.c：

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <rdma/rdma_cma.h>
?
#define TEST_NZ(x) do { if ( (x)) die("error: " #x " failed (returned non-zero)." ); } while (0)
#define TEST_Z(x) ?do { if (!(x)) die("error: " #x " failed (returned zero/null)."); } while (0)
?
const int BUFFER_SIZE = 1024;
?
struct context {
? struct ibv_context *ctx;
? struct ibv_pd *pd;
? struct ibv_cq *cq;
? struct ibv_comp_channel *comp_channel;
?
? pthread_t cq_poller_thread;
};
?
struct connection {
? struct ibv_qp *qp;
?
? struct ibv_mr *recv_mr;
? struct ibv_mr *send_mr;
?
? char *recv_region;
? char *send_region;
};
?
static void die(const char *reason);
?
static void build_context(struct ibv_context *verbs);
static void build_qp_attr(struct ibv_qp_init_attr *qp_attr);
static void * poll_cq(void *);
static void post_receives(struct connection *conn);
static void register_memory(struct connection *conn);
?
static void on_completion(struct ibv_wc *wc);
static int on_connect_request(struct rdma_cm_id *id);
static int on_connection(void *context);
static int on_disconnect(struct rdma_cm_id *id);
static int on_event(struct rdma_cm_event *event);
?
static struct context *s_ctx = NULL;
?
int main(int argc, char **argv)
{
#if _USE_IPV6
? struct sockaddr_in6 addr;
#else
? struct sockaddr_in addr;
#endif
? struct rdma_cm_event *event = NULL;
? struct rdma_cm_id *listener = NULL;
? struct rdma_event_channel *ec = NULL;
? uint16_t port = 0;
?
? memset(&addr, 0, sizeof(addr));
#if _USE_IPV6
? addr.sin6_family = AF_INET6;
#else
? addr.sin_family = AF_INET;
#endif
?
? TEST_Z(ec = rdma_create_event_channel());
? TEST_NZ(rdma_create_id(ec, &listener, NULL, RDMA_PS_TCP));
? TEST_NZ(rdma_bind_addr(listener, (struct sockaddr *)&addr));
? TEST_NZ(rdma_listen(listener, 10)); /* backlog=10 is arbitrary */
?
? port = ntohs(rdma_get_src_port(listener)); //rdma_get_src_port 返回listener對(duì)應(yīng)的tcp 端口
?
? printf("listening on port %d.\n", port);
?
? while (rdma_get_cm_event(ec, &event) == 0) {
? ? struct rdma_cm_event event_copy;
?
? ? memcpy(&event_copy, event, sizeof(*event));
? ? rdma_ack_cm_event(event);
?
? ? if (on_event(&event_copy))
? ? ? break;
? }
?
? rdma_destroy_id(listener);
? rdma_destroy_event_channel(ec);
?
? return 0;
}
?
void die(const char *reason)
{
? fprintf(stderr, "%s\n", reason);
? exit(EXIT_FAILURE);
}
?
void build_context(struct ibv_context *verbs)
{
? if (s_ctx) {
? ? if (s_ctx->ctx != verbs)
? ? ? die("cannot handle events in more than one context.");
?
? ? return;
? }
?
? s_ctx = (struct context *)malloc(sizeof(struct context));
?
? s_ctx->ctx = verbs;
?
? TEST_Z(s_ctx->pd = ibv_alloc_pd(s_ctx->ctx));
? TEST_Z(s_ctx->comp_channel = ibv_create_comp_channel(s_ctx->ctx));
? TEST_Z(s_ctx->cq = ibv_create_cq(s_ctx->ctx, 10, NULL, s_ctx->comp_channel, 0)); /* cqe=10 is arbitrary */
? TEST_NZ(ibv_req_notify_cq(s_ctx->cq, 0)); #完成完成隊(duì)列與完成通道的關(guān)聯(lián)
?
? TEST_NZ(pthread_create(&s_ctx->cq_poller_thread, NULL, poll_cq, NULL));
}
?
void build_qp_attr(struct ibv_qp_init_attr *qp_attr)
{
? memset(qp_attr, 0, sizeof(*qp_attr));
?
? qp_attr->send_cq = s_ctx->cq;
? qp_attr->recv_cq = s_ctx->cq;
? qp_attr->qp_type = IBV_QPT_RC;
?
? qp_attr->cap.max_send_wr = 10;
? qp_attr->cap.max_recv_wr = 10;
? qp_attr->cap.max_send_sge = 1;
? qp_attr->cap.max_recv_sge = 1;
}
?
void * poll_cq(void *ctx)
{
? struct ibv_cq *cq;
? struct ibv_wc wc;
?
? while (1) {
? ? TEST_NZ(ibv_get_cq_event(s_ctx->comp_channel, &cq, &ctx));
? ? ibv_ack_cq_events(cq, 1);
? ? TEST_NZ(ibv_req_notify_cq(cq, 0));
?
? ? while (ibv_poll_cq(cq, 1, &wc))
? ? ? on_completion(&wc);
? }
?
? return NULL;
}
?
void post_receives(struct connection *conn)
{
? struct ibv_recv_wr wr, *bad_wr = NULL;
? struct ibv_sge sge;
?
? wr.wr_id = (uintptr_t)conn;
? wr.next = NULL;
? wr.sg_list = &sge;
? wr.num_sge = 1;
?
? sge.addr = (uintptr_t)conn->recv_region;
? sge.length = BUFFER_SIZE;
? sge.lkey = conn->recv_mr->lkey;
?
? TEST_NZ(ibv_post_recv(conn->qp, &wr, &bad_wr));
}
?
void register_memory(struct connection *conn)
{
? conn->send_region = malloc(BUFFER_SIZE);
? conn->recv_region = malloc(BUFFER_SIZE);
?
? TEST_Z(conn->send_mr = ibv_reg_mr(
? ? s_ctx->pd,
? ? conn->send_region,
? ? BUFFER_SIZE,
? ? 0));
?
? TEST_Z(conn->recv_mr = ibv_reg_mr(
? ? s_ctx->pd,
? ? conn->recv_region,
? ? BUFFER_SIZE,
? ? IBV_ACCESS_LOCAL_WRITE));
}
?
void on_completion(struct ibv_wc *wc)
{
? if (wc->status != IBV_WC_SUCCESS)
? ? die("on_completion: status is not IBV_WC_SUCCESS.");
?
? if (wc->opcode & IBV_WC_RECV) {
? ? struct connection *conn = (struct connection *)(uintptr_t)wc->wr_id;
?
? ? printf("received message: %s\n", conn->recv_region);
?
? } else if (wc->opcode == IBV_WC_SEND) {
? ? printf("send completed successfully.\n");
? }
}
?
int on_connect_request(struct rdma_cm_id *id)
{
? struct ibv_qp_init_attr qp_attr;
? struct rdma_conn_param cm_params;
? struct connection *conn;
?
? printf("received connection request.\n");
?
? build_context(id->verbs);
? build_qp_attr(&qp_attr);
?
? TEST_NZ(rdma_create_qp(id, s_ctx->pd, &qp_attr));
?
? id->context = conn = (struct connection *)malloc(sizeof(struct connection));
? conn->qp = id->qp;
?
? register_memory(conn);
? post_receives(conn);
?
? memset(&cm_params, 0, sizeof(cm_params));
? TEST_NZ(rdma_accept(id, &cm_params));
?
? return 0;
}
?
int on_connection(void *context)
{
? struct connection *conn = (struct connection *)context;
? struct ibv_send_wr wr, *bad_wr = NULL;
? struct ibv_sge sge;
?
? snprintf(conn->send_region, BUFFER_SIZE, "message from passive/server side with pid %d", getpid());
?
? printf("connected. posting send...\n");
?
? memset(&wr, 0, sizeof(wr));
?
? wr.opcode = IBV_WR_SEND;
? wr.sg_list = &sge;
? wr.num_sge = 1;
? wr.send_flags = IBV_SEND_SIGNALED;
?
? sge.addr = (uintptr_t)conn->send_region;
? sge.length = BUFFER_SIZE;
? sge.lkey = conn->send_mr->lkey;
?
? TEST_NZ(ibv_post_send(conn->qp, &wr, &bad_wr));
?
? return 0;
}
?
int on_disconnect(struct rdma_cm_id *id)
{
? struct connection *conn = (struct connection *)id->context;
?
? printf("peer disconnected.\n");
?
? rdma_destroy_qp(id);
?
? ibv_dereg_mr(conn->send_mr);
? ibv_dereg_mr(conn->recv_mr);
?
? free(conn->send_region);
? free(conn->recv_region);
?
? free(conn);
?
? rdma_destroy_id(id);
?
? return 0;
}
?
int on_event(struct rdma_cm_event *event)
{
? int r = 0;
?
? if (event->event == RDMA_CM_EVENT_CONNECT_REQUEST)
? ? r = on_connect_request(event->id);
? else if (event->event == RDMA_CM_EVENT_ESTABLISHED)
? ? r = on_connection(event->id->context);
? else if (event->event == RDMA_CM_EVENT_DISCONNECTED)
? ? r = on_disconnect(event->id);
? else
? ? die("on_event: unknown event.");
?
? return r;
}

客戶端client.c：

#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <rdma/rdma_cma.h>
?
#define TEST_NZ(x) do { if ( (x)) die("error: " #x " failed (returned non-zero)." ); } while (0)
#define TEST_Z(x) ?do { if (!(x)) die("error: " #x " failed (returned zero/null)."); } while (0)
?
const int BUFFER_SIZE = 1024;
const int TIMEOUT_IN_MS = 500; /* ms */
?
struct context {
? struct ibv_context *ctx;
? struct ibv_pd *pd;
? struct ibv_cq *cq;
? struct ibv_comp_channel *comp_channel;
?
? pthread_t cq_poller_thread;
};
?
struct connection {
? struct rdma_cm_id *id;
? struct ibv_qp *qp;
?
? struct ibv_mr *recv_mr;
? struct ibv_mr *send_mr;
?
? char *recv_region;
? char *send_region;
?
? int num_completions;
};
?
static void die(const char *reason);
?
static void build_context(struct ibv_context *verbs);
static void build_qp_attr(struct ibv_qp_init_attr *qp_attr);
static void * poll_cq(void *);
static void post_receives(struct connection *conn);
static void register_memory(struct connection *conn);
?
static int on_addr_resolved(struct rdma_cm_id *id);
static void on_completion(struct ibv_wc *wc);
static int on_connection(void *context);
static int on_disconnect(struct rdma_cm_id *id);
static int on_event(struct rdma_cm_event *event);
static int on_route_resolved(struct rdma_cm_id *id);
?
static struct context *s_ctx = NULL;
?
int main(int argc, char **argv)
{
? struct addrinfo *addr;
? struct rdma_cm_event *event = NULL;
? struct rdma_cm_id *conn= NULL;
? struct rdma_event_channel *ec = NULL;
?
? if (argc != 3)
? ? die("usage: client <server-address> <server-port>");
?
? TEST_NZ(getaddrinfo(argv[1], argv[2], NULL, &addr));
?
? TEST_Z(ec = rdma_create_event_channel());
? TEST_NZ(rdma_create_id(ec, &conn, NULL, RDMA_PS_TCP));
? TEST_NZ(rdma_resolve_addr(conn, NULL, addr->ai_addr, TIMEOUT_IN_MS));
?
? freeaddrinfo(addr);
?
? while (rdma_get_cm_event(ec, &event) == 0) {
? ? struct rdma_cm_event event_copy;
?
? ? memcpy(&event_copy, event, sizeof(*event));
? ? rdma_ack_cm_event(event);
?
? ? if (on_event(&event_copy))
? ? ? break;
? }
?
? rdma_destroy_event_channel(ec);
?
? return 0;
}
?
void die(const char *reason)
{
? fprintf(stderr, "%s\n", reason);
? exit(EXIT_FAILURE);
}
?
void build_context(struct ibv_context *verbs)
{
? if (s_ctx) {
? ? if (s_ctx->ctx != verbs)
? ? ? die("cannot handle events in more than one context.");
?
? ? return;
? }
?
? s_ctx = (struct context *)malloc(sizeof(struct context));
?
? s_ctx->ctx = verbs;
?
? TEST_Z(s_ctx->pd = ibv_alloc_pd(s_ctx->ctx));
? TEST_Z(s_ctx->comp_channel = ibv_create_comp_channel(s_ctx->ctx));
? TEST_Z(s_ctx->cq = ibv_create_cq(s_ctx->ctx, 10, NULL, s_ctx->comp_channel, 0)); /* cqe=10 is arbitrary */
? TEST_NZ(ibv_req_notify_cq(s_ctx->cq, 0));
?
? TEST_NZ(pthread_create(&s_ctx->cq_poller_thread, NULL, poll_cq, NULL));
}
?
void build_qp_attr(struct ibv_qp_init_attr *qp_attr)
{
? memset(qp_attr, 0, sizeof(*qp_attr));
?
? qp_attr->send_cq = s_ctx->cq;
? qp_attr->recv_cq = s_ctx->cq;
? qp_attr->qp_type = IBV_QPT_RC;
?
? qp_attr->cap.max_send_wr = 10;
? qp_attr->cap.max_recv_wr = 10;
? qp_attr->cap.max_send_sge = 1;
? qp_attr->cap.max_recv_sge = 1;
}
?
void * poll_cq(void *ctx)
{
? struct ibv_cq *cq;
? struct ibv_wc wc;
?
? while (1) {
? ? TEST_NZ(ibv_get_cq_event(s_ctx->comp_channel, &cq, &ctx));
? ? ibv_ack_cq_events(cq, 1);
? ? TEST_NZ(ibv_req_notify_cq(cq, 0));
?
? ? while (ibv_poll_cq(cq, 1, &wc))
? ? ? on_completion(&wc);
? }
?
? return NULL;
}
?
void post_receives(struct connection *conn)
{
? struct ibv_recv_wr wr, *bad_wr = NULL;
? struct ibv_sge sge;
?
? wr.wr_id = (uintptr_t)conn;
? wr.next = NULL;
? wr.sg_list = &sge;
? wr.num_sge = 1;
?
? sge.addr = (uintptr_t)conn->recv_region;
? sge.length = BUFFER_SIZE;
? sge.lkey = conn->recv_mr->lkey;
?
? TEST_NZ(ibv_post_recv(conn->qp, &wr, &bad_wr));
}
?
void register_memory(struct connection *conn)
{
? conn->send_region = malloc(BUFFER_SIZE);
? conn->recv_region = malloc(BUFFER_SIZE);
?
? TEST_Z(conn->send_mr = ibv_reg_mr(
? ? s_ctx->pd,?
? ? conn->send_region,?
? ? BUFFER_SIZE,?
? ? 0));
?
? TEST_Z(conn->recv_mr = ibv_reg_mr(
? ? s_ctx->pd,?
? ? conn->recv_region,?
? ? BUFFER_SIZE,?
? ? IBV_ACCESS_LOCAL_WRITE));
}
?
int on_addr_resolved(struct rdma_cm_id *id)
{
? struct ibv_qp_init_attr qp_attr;
? struct connection *conn;
?
? printf("address resolved.\n");
?
? build_context(id->verbs);
? build_qp_attr(&qp_attr);
?
? TEST_NZ(rdma_create_qp(id, s_ctx->pd, &qp_attr));
?
? id->context = conn = (struct connection *)malloc(sizeof(struct connection));
?
? conn->id = id;
? conn->qp = id->qp;
? conn->num_completions = 0;
?
? register_memory(conn);
? post_receives(conn);
?
? TEST_NZ(rdma_resolve_route(id, TIMEOUT_IN_MS));
?
? return 0;
}
?
void on_completion(struct ibv_wc *wc)
{
? struct connection *conn = (struct connection *)(uintptr_t)wc->wr_id;
?
? if (wc->status != IBV_WC_SUCCESS)
? ? die("on_completion: status is not IBV_WC_SUCCESS.");
?
? if (wc->opcode & IBV_WC_RECV)
? ? printf("received message: %s\n", conn->recv_region);
? else if (wc->opcode == IBV_WC_SEND)
? ? printf("send completed successfully.\n");
? else
? ? die("on_completion: completion isn't a send or a receive.");
?
? if (++conn->num_completions == 2)
? ? rdma_disconnect(conn->id);
}
?
int on_connection(void *context)
{
? struct connection *conn = (struct connection *)context;
? struct ibv_send_wr wr, *bad_wr = NULL;
? struct ibv_sge sge;
?
? snprintf(conn->send_region, BUFFER_SIZE, "message from active/client side with pid %d", getpid());
?
? printf("connected. posting send...\n");
?
? memset(&wr, 0, sizeof(wr));
?
? wr.wr_id = (uintptr_t)conn;
? wr.opcode = IBV_WR_SEND;
? wr.sg_list = &sge;
? wr.num_sge = 1;
? wr.send_flags = IBV_SEND_SIGNALED;
?
? sge.addr = (uintptr_t)conn->send_region;
? sge.length = BUFFER_SIZE;
? sge.lkey = conn->send_mr->lkey;
?
? TEST_NZ(ibv_post_send(conn->qp, &wr, &bad_wr));
?
? return 0;
}
?
int on_disconnect(struct rdma_cm_id *id)
{
? struct connection *conn = (struct connection *)id->context;
?
? printf("disconnected.\n");
?
? rdma_destroy_qp(id);
?
? ibv_dereg_mr(conn->send_mr);
? ibv_dereg_mr(conn->recv_mr);
?
? free(conn->send_region);
? free(conn->recv_region);
?
? free(conn);
?
? rdma_destroy_id(id);
?
? return 1; /* exit event loop */
}
?
int on_event(struct rdma_cm_event *event)
{
? int r = 0;
?
? if (event->event == RDMA_CM_EVENT_ADDR_RESOLVED)
? ? r = on_addr_resolved(event->id);
? else if (event->event == RDMA_CM_EVENT_ROUTE_RESOLVED)
? ? r = on_route_resolved(event->id);
? else if (event->event == RDMA_CM_EVENT_ESTABLISHED)
? ? r = on_connection(event->id->context);
? else if (event->event == RDMA_CM_EVENT_DISCONNECTED)
? ? r = on_disconnect(event->id);
? else
? ? die("on_event: unknown event.");
?
? return r;
}
?
int on_route_resolved(struct rdma_cm_id *id)
{
? struct rdma_conn_param cm_params;
?
? printf("route resolved.\n");
?
? memset(&cm_params, 0, sizeof(cm_params));
? TEST_NZ(rdma_connect(id, &cm_params));
?
? return 0;
}

更多講解教程
InfiniBand, Verbs, RDMA | https://thegeekinthecorner.wordpress.com/category/infiniband-verbs-rdma/

RDMA read and write with IB verbs | https://thegeekinthecorner.wordpress.com/2010/09/28/rdma-read-and-write-with-ib-verbs/

http://www.hpcadvisorycouncil.com/pdf/building-an-rdma-capable-application-with-ib-verbs.pdf文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-757090.html

到了這里，關(guān)于RDMA編程實(shí)例rdma_cm API的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！