描述
ibv_modify_qp()修改队列对的属性。
更改的属性描述了QP的发送和接收属性。在UC和RC QP中，这意味着将QP与远程QP连接。
在Infiniband中，其中一个应该执行到子网管理员(SA)的路径查询(one should perform path query to the Subnet Administrator (SA))，以便确定应该为QP配置哪些属性或作为最佳解决方案，请使用Communication Manager(CM)或通用RDMA CM代理(CMA)连接QP。但是，有些应用程序倾向于自己连接QP，并通过sockets交换数据来决定要使用哪些QP属性。
在RoCE中，其中一个应该为连接的QP配置在QP属性中的GRH，或者为UD QP配置在地址句柄(Address Handle中)的GRH。
在iWARP中，只能使用通用RDMA CM代理(CMA)连接QP。
struct ibv_qp_attr描述了队列对的属性。

struct ibv_qp_attr 描述QP的属性

struct ibv_qp_attr {enum ibv_qp_state qp_state;enum ibv_qp_state  cur_qp_state;enum ibv_mtu       path_mtu;enum ibv_mig_state path_mig_state;uint32_t     qkey;uint32_t       rq_psn;uint32_t     sq_psn;uint32_t     dest_qp_num;int         qp_access_flags;struct ibv_qp_cap   cap;struct ibv_ah_attr  ah_attr;struct ibv_ah_attr  alt_ah_attr;uint16_t        pkey_index;uint16_t     alt_pkey_index;uint8_t          en_sqd_async_notify;uint8_t         sq_draining;uint8_t         max_rd_atomic;uint8_t           max_dest_rd_atomic;uint8_t          min_rnr_timer;uint8_t           port_num;uint8_t            timeout;uint8_t         retry_cnt;uint8_t           rnr_retry;uint8_t           alt_port_num;uint8_t            alt_timeout;
};

struct ibv_qp_attr的完整说明

qp_state	The next QP state. It can be one of the following enumerated values: IBV_QPS_RESET - Reset state IBV_QPS_INIT - Initialized state IBV_QPS_RTR - Ready To Receive state IBV_QPS_RTS - Ready To Send state IBV_QPS_SQD - Send Queue Drain state IBV_QPS_SQE - Send Queue Error state IBV_QPS_ERR - Error state
cur_qp_state	Assume that this is the current QP state. This is useful if it is known to the application that the QP state is different from the assumed state by the low-level driver. It can be one of the enumerated values as qp_state 假设这是当前的QP状态。如果应用程序知道QP状态与低级驱动程序假定的状态不同，这将很有用。它可以是qp_state枚举值之一
path_mtu	The path MTU (Maximum Transfer Unit) i.e. the maximum payload size of a packet that can be transferred in the path. It can be one of the following enumerated values: IBV_MTU_256 - 256 bytes IBV_MTU_512 - 512 bytes IBV_MTU_1024 - 1024 bytes IBV_MTU_2048 - 2048 bytes IBV_MTU_4096 - 4096 bytes For UC and RC QPs, when needed, the RDMA device will automatically fragment the messages to packet of this size.
path_mig_state	The state of the QP's path migration state machine if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags). It can be one of the following enumerated values: IBV_MIG_MIGRATED - The state machine of path migration is Migrated, i.e. initial state of migration was done IBV_MIG_REARM - The state machine of path migration is Rearm, i.e. attempt to try to coordinate the remote RC QP to move both local and remote QPs to Armed state IBV_MIG_ARMED - The state machine of path migration is Armed, i.e. both local and remote QPs are ready to perform a path migration
qkey	The Q_Key that incoming messages are check against and possibly used as the outgoing Q_Key (if the MSB of the q_key in the Send Request is set). Relevant only for UD QPs
rq_psn	A 24 bits value of the Packet Sequence Number of the received packets for RC and UC QPs
sq_psn	A 24 bits value of the Packet Sequence Number of the sent packets for any QP
dest_qp_num	A 24 bits value of the remote QP number of RC and UC QPs; when sending data, packets will be sent to this QP number and when receiving data, packets will be accepted only from this QP number
qp_access_flags	Allowed access flags of the remote operations for incoming packets of RC and UC QPs. It is either 0 or the bitwise OR of one or more of the following flags: IBV_ACCESS_REMOTE_WRITE - Allow incoming RDMA Writes on this QP IBV_ACCESS_REMOTE_READ - Allow incoming RDMA Reads on this QP IBV_ACCESS_REMOTE_ATOMIC - Allow incoming Atomic operations on this QP
cap	New attributes to the number of Wore Requests in the Queue Pair, as described in the table below, if resize QP is supported by the device (IBV_DEVICE_RESIZE_MAX_WR is set in dev_cap.device_cap_flags). Upon a successful Queue Pair resize, this structure will hold the actual Queue Pair attributes
ah_attr	Address vector of the primary path which describes the path information to the remote QP as described in the table below
alt_ah_attr	Address vector of the alternate path which describes the path information to the remote QP as described in the table below. Can be used only if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags)
pkey_index	Primary P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Primary path
alt_pkey_index	Alternate P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Alternate path
en_sqd_async_notify	If non-zero, generate the affiliated asynchronous event IBV_EVENT_SQ_DRAINED when the QP state becomes SQD.drained, i.e. the Send Queue is drained
sq_draining	If set, indication that Send Queue draining is in progress. Not relevant for ibv_modify_qp()
max_rd_atomic	The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as an initiator. Relevant only for RC QPs
max_dest_rd_atomic	The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as a destination. Relevant only for RC QPs
min_rnr_timer	Minimum RNR NAK Timer Field Value. When an incoming message to this QP should consume a Work Request from the Receive Queue, but not Work Request is outstanding on that Queue, the QP will send an RNR NAK packet to the initiator. It does not affect RNR NAKs sent for other reasons. The value can be one of the following numeric values since those values aren’t enumerated: 0 - 655.36 milliseconds delay 1 - 0.01 milliseconds delay 2 - 0.02 milliseconds delay 3 - 0.03 milliseconds delay 4 - 0.04 milliseconds delay 5 - 0.06 milliseconds delay 6 - 0.08 milliseconds delay 7 - 0.12 milliseconds delay 8 - 0.16 milliseconds delay 9 - 0.24 milliseconds delay 10 - 0.32 milliseconds delay 11 - 0.48 milliseconds delay 12 - 0.64 milliseconds delay 13 - 0.96 milliseconds delay 14 - 1.28 milliseconds delay 15 - 1.92 milliseconds delay 16 - 2.56 milliseconds delay 17 - 3.84 milliseconds delay 18 - 5.12 milliseconds delay 19 - 7.68 milliseconds delay 20 - 10.24 milliseconds delay 21 - 15.36 milliseconds delay 22 - 20.48 milliseconds delay 23 - 30.72 milliseconds delay 24 - 40.96 milliseconds delay 25 - 61.44 milliseconds delay 26 - 81.92 milliseconds delay 27 - 122.88 milliseconds delay 28 - 163.84 milliseconds delay 29 - 245.76 milliseconds delay 30 - 327.68 milliseconds delay 31 - 491.52 milliseconds delay Relevant only for RC QPs
port_num	Primary physical port number associated with this QP
timeout	The minimum timeout that a QP waits for ACK/NACK from remote QP before retransmitting the packet. The value zero is special value which means wait an infinite time for the ACK/NACK (useful for debugging). For any other value of timeout, the time calculation is:  usec. For your convenient, here is the summary of each value and its timeout: 0 - infinite 1 - 8.192 usec (0.000008 sec) 2 - 16.384 usec (0.000016 sec) 3 - 32.768 usec (0.000032 sec) 4 - 65.536 usec (0.000065 sec) 5 - 131.072 usec (0.000131 sec) 6 - 262.144 usec (0.000262 sec) 7 - 524.288 usec (0.000524 sec) 8 - 1048.576 usec (0.00104 sec) 9 - 2097.152 usec (0.00209 sec) 10 - 4194.304 usec (0.00419 sec) 11 - 8388.608 usec (0.00838 sec) 12 - 16777.22 usec (0.01677 sec) 13 - 33554.43 usec (0.0335 sec) 14 - 67108.86 usec (0.0671 sec) 15 - 134217.7 usec (0.134 sec) 16 - 268435.5 usec (0.268 sec) 17 - 536870.9 usec (0.536 sec) 18 - 1073742 usec (1.07 sec) 19 - 2147484 usec (2.14 sec) 20 - 4294967 usec (4.29 sec) 21 - 8589935 usec (8.58 sec) 22 - 17179869 usec (17.1 sec) 23 - 34359738 usec (34.3 sec) 24 - 68719477 usec (68.7 sec) 25 - 137000000 usec (137 sec) 26 - 275000000 usec (275 sec) 27 - 550000000 usec (550 sec) 28 - 1100000000 usec (1100 sec) 29 - 2200000000 usec (2200 sec) 30 - 4400000000 usec (4400 sec) 31 - 8800000000 usec (8800 sec) Relevant only to RC QPs
retry_cnt	A 3 bits value of the total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the primary path
rnr_retry	A 3 bits value of the total number of times that the QP will try to resend the packets when an RNR NACK was sent by the remote QP before reporting an error. The value 7 is special and specify to retry infinite times in case of RNR
alt_port_num	Alternate physical port number associated with this QP
alt_timeout	The total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the alternate path

关于QP属性中某些特定值的一些警告：

• 在开发过程中，将attr.retry_cnt和attr.rnr_retry设置为0是一个好习惯。如果代码中存在竞争，最好在开发阶段检测它们
• attr.timeout中的值0表示一直等待ACK或NACK。这意味着，如果消息中的任何数据包丢失并且没有发送ACK或NACK，则不会发生任何重试，并且QP只会停止发送数据
• attr.rnr_retry中的值7表示在远程端发送RNR Nack时，发送消息重试次数无上限

struct ibv_qp_cap 描述QP的size(用于发送和接收队列)。

struct ibv_qp_cap {uint32_t      max_send_wr;uint32_t        max_recv_wr;uint32_t        max_send_sge;uint32_t       max_recv_sge;uint32_t       max_inline_data;
};

struct ibv_qp_cap: 说明

struct ibv_ah_attr 描述了QP的地址向量

struct ibv_ah_attr {struct ibv_global_route grh;uint16_t        dlid;uint8_t            sl;uint8_t          src_path_bits;uint8_t           static_rate;uint8_t         is_global;uint8_t           port_num;
};

struct ibv_ah_attr:说明

struct ibv_global_route   AH中描述将要发送的数据包的GRH中使用的值

struct ibv_global_route {union ibv_gid        dgid;uint32_t       flow_label;uint8_t          sgid_index;uint8_t          hop_limit;uint8_t           traffic_class;
};

struct ibv_global_route属性说明:

attr_mask specifies the QP attributes to be modified. It is either 0 or the bitwise OR of one or more of the following flags:

attr_mask指定要修改QP的哪些属性。它是0或以下一个或多个标志的按位或：

The following table specify the full supported transition for QP with service type IBV_QPT_UD and the attributes it accepts:

下表指定了服务类型为IBV_QPT_UD的QP的支持的状态转换及其接受的属性：

The following table specify the full supported transition for QP with service type IBV_QPT_UC and the attributes it accepts:

下表指定了服务类型为IBV_QPT_UD的QP的支持的状态转换及其接受的属性：

The following table specify the full supported transition for QP with service type IBV_QPT_RC and the attributes it accepts:

下表指定了服务类型为IBV_QPT_RC 的QP的支持的状态转换及其接受的属性：

参数

返回值

示例

1) Modify a UD QP from the RESET state to the RTS state:

将UD QP从RESET状态修改为RTS状态：

/* this example assumed that the variables my_port, my_psn are declared and initialized with valid values */
此示例假定变量my_port，my_psn均已声明并使用有效值初始化
struct ibv_qp *qp;
struct ibv_qp_attr attr;memset(&attr, 0, sizeof(attr));attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qkey            = 0x22222222;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE      |IBV_QP_PKEY_INDEX |IBV_QP_PORT       |IBV_QP_QKEY)) {fprintf(stderr, "Failed to modify QP to INIT\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTR;if (ibv_modify_qp(qp, &attr, IBV_QP_STATE)) {fprintf(stderr, "Failed to modify QP to RTR\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTS;
attr.sq_psn     = my_psn;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE |IBV_QP_SQ_PSN)) {fprintf(stderr, "Failed to modify QP to RTS\n");return 1;
}

2) Modify a UC QP from the RESET state to the RTS state:

将UC QP从RESET状态修改为RTS状态：

/* this example assumed that the variables my_port, my_psn, my_mtu, my_sl, remote_qpn, remote_psn, remote_lid are declared and initialized with valid values */
struct ibv_qp *qp;
struct ibv_qp_attr attr;memset(&attr, 0, sizeof(attr));attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qp_access_flags = 0;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE        |IBV_QP_PKEY_INDEX   |IBV_QP_PORT         |IBV_QP_ACCESS_FLAGS)) {fprintf(stderr, "Failed to modify QP to INIT\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTR;
attr.path_mtu       = my_mtu;
attr.dest_qp_num    = remote_qpn;
attr.rq_psn     = remote_psn;
attr.ah_attr.is_global     = 0;
attr.ah_attr.dlid      = remote_lid;
attr.ah_attr.sl        = my_sl;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num      = my_port;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE    |IBV_QP_AV       |IBV_QP_PATH_MTU |IBV_QP_DEST_QPN |IBV_QP_RQ_PSN)) {fprintf(stderr, "Failed to modify QP to RTR\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTS;
attr.sq_psn     = my_psn;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE  |IBV_QP_SQ_PSN)) {fprintf(stderr, "Failed to modify QP to RTS\n");return 1;
}

3) Modify an RC QP from the RESET state to the RTS state:

将RC QP从RESET状态修改为RTS状态：

/* this example assumed that the variables my_port, my_psn, my_mtu, my_sl, remote_qpn, remote_psn, remote_lid are declared and initialized with valid values */
struct ibv_qp *qp;
struct ibv_qp_attr attr;memset(&attr, 0, sizeof(attr));attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qp_access_flags = 0;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE      |IBV_QP_PKEY_INDEX |IBV_QP_PORT       |IBV_QP_ACCESS_FLAGS)) {fprintf(stderr, "Failed to modify QP to INIT\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTR;
attr.path_mtu       = my_mtu;
attr.dest_qp_num    = remote_qpn;
attr.rq_psn     = remote_psn;
attr.max_dest_rd_atomic = 1;
attr.min_rnr_timer  = 12;
attr.ah_attr.is_global     = 0;
attr.ah_attr.dlid      = remote_lid;
attr.ah_attr.sl        = my_sl;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num      = my_port;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE              |IBV_QP_AV                 |IBV_QP_PATH_MTU           |IBV_QP_DEST_QPN           |IBV_QP_RQ_PSN             |IBV_QP_MAX_DEST_RD_ATOMIC |IBV_QP_MIN_RNR_TIMER)) {fprintf(stderr, "Failed to modify QP to RTR\n");return -1;
}memset(&attr, 0, sizeof(attr));attr.qp_state       = IBV_QPS_RTS;
attr.sq_psn     = my_psn;
attr.timeout        = 14;
attr.retry_cnt      = 7;
attr.rnr_retry      = 7; /* infinite */
attr.max_rd_atomic  = 1;if (ibv_modify_qp(qp, &attr,IBV_QP_STATE              |IBV_QP_TIMEOUT            |IBV_QP_RETRY_CNT          |IBV_QP_RNR_RETRY          |IBV_QP_SQ_PSN             |IBV_QP_MAX_QP_RD_ATOMIC)) {fprintf(stderr, "Failed to modify QP to RTS\n");return 1;
}

常见问题

为什么需要调用ibv_modify_qp()？
您需要调用ibv_modify_qp()才能将QP修改为可以接收和发送数据的状态

我可以将QP移到RTR状态并保持在该状态吗？
是的你可以。如果QP只是接收者

我如何知道要配置QP的哪些属性？

In Infiniband, you should perform path query to the SA or use CM or CMA, in iWARP, you should use the CMA
在Infiniband中，应该执行到SA的路径查询，或者使用CM或CMA；在iWARP中，应该使用CMA。

ibv_modify_qp()失败了，该QP的属性现在是什么？
与调用ibv_modify_qp()之前完全相同，未对QP进行任何更改。尽管该失败可能是由于某个错误导致RDMA设备将QP移至SQE或ERROR状态引起的

ibv_modify_qp()在通过RoCE已连接QP 进行INIT-> RTR转换时失败，这是什么原因？

使用RoCE时，必须配置GRH。
GRH在连接的QP中：作为QP属性的一部分
GRH在UD QP中：作为“地址句柄”属性的一部分

1. Igor R. says:February 11, 2014

如果您能阐明以下主题，我将不胜感激。
根据“ Infinibeand网络体系结构”(*参见下面的引用)，在RDMA读取request期间，允许重试时从内存中重新读取响应者的RQ logic 。是否可以通过设置retry_cnt = rnr_cnt = 0来完全禁用此功能？
换句话说，可以确保在这种情况下，单个RDMA 读取请求 永远不会导致对任何远程位置的双重读取吗？ (如果这是设备内存，则这可能至关重要，并且访问可能会产生副作用。)

(I'd appreciate if you could shed some light on the following subject.
As per "Infinibeand Network Architecture"(* see quotations below), during RDMA Read request, the responder's RQ logic is allowed to re-read from memory when re-trying. Can we totally disable this functionality by setting retry_cnt=rnr_cnt=0?
In other words, can one be sure in such a case, that a single RDMA Read request will never cause double read of any remote location? (This can be crucial if it's a device memory, and an access might have side effects.

Thanks!)

* Infiniband Network Architecture, p.34:(Infiniband网络体系结构，第34页：)
<<
This service is referred to as "reliable" for the following reasons:
-- The destination QP's RQ Logic verifies the PSN in each request packet to ensure that all of the message's request packets are received in order, that none are missing, and, if any duplicate request packets are received, that they are only processed once. There is one exception; upon receipt of a duplicate memory read, the data is read from memory again.

>>

And in details (p.394):(还有详细说明(第394页)：)
<<
Effects of Retry on Requester and Responder The following list defines the sequence of events:

6. RQ Logic receives duplicate copy of request. When the second copy (the retried one) of the request packet eventually arrives at the RQ Logic, it has the same PSN as the original request packet. Its PSN, therefore, falls into the Ack’ d PSN region (also referred to as the Duplicate region), thereby identifying it as a duplicate request packet. The actions taken by the RQ Logic are determined by the request type:

— If it’ s a duplicate RDMA Read:
– The RQ Logic re-executes the read from its local memory.
– It sends back the requested read data in a series of one or more RDMA Read response packets.
– The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
– The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
– After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN

.Reply Igor R. says:February 12, 2014

好吧，这里的重试计数器似乎是“红鲱鱼”，因为真正的问题是远程HCA无法缓冲读取的数据。因此，在发生故障的情况下，即使不会发生自动重传，也将需要手动进行一次-以便无论如何将数据读取两次。因此，仅禁用自动重试是不够的，并且需要更复杂的方法。

• Well, it seems that the retry counter is a "red herring" here, as the real problem is that remote HCA doesn't buffer the read data. So, in case of failure, even if the automatic retransmission won't occur, a manual one will be required - so that the data will be read twice anyway. Thus, solely disabling automatic retries is not sufficient, and more complicated approach is needed.

Reply Dotan Barak says:February 12, 2014

• 我必须承认，我不太了解您要执行的操作...
  是的，如果retry_cnt为0，则不会发生重传，
  QP将在请求方进入错误状态。
  这是您所需的预期行为吗？
  如果您必须从一侧读取数据而无需从设备读取任何数据，
  也许RDMA读取不是正确的方法
  (也许将数据缓存在本地内存中并执行RDMA写操作会更好。)
  谢谢
  多坦

I must admit that I don't really understand what you are trying to perform...

Yes, if the retry_cnt will be 0 retransmission won't happen,
and the QP will get into the error state in the requester side.

Is this is the expected behavior that you need?
If you must read the data from one side without any read from the device,
maybe RDMA Read isn't the right approach
(maybe cache the data in local memory and perform RDMA Write is better..)

Thanks
Dotan

• Igor R. says:February 12, 2014

• 感谢您回复。
  在我的情况下，响应方(设备所在的位置)必须尽可能保持被动状态。我只是想考虑各种可能性并了解其后果...

  Thanks for the input.
  In my case the responding side (where the device is) must remain as passive as possible. I'm just trying to consider various possibilities and to understand their consequences...

1. Alan says:August 29, 2014

   Hi Dotan,

   Do we have to memset the ibv_qp_attr structure before using it in the ibv_modify_qp() function? I am thinking there are flags in the function API to indicate which fields are being used why we still need to memset it?

   Thanks.

   Reply

2. Dotan Barak says:August 29, 2014

   • Hi Alan.

     In general - you are correct (and in most cases it will work without any problem).

     However:
     * Most ibv_* structu don't have bitmask to specify the valid attributes
     * The struct ibv_qp_attr *may* be extended (although the developers try hard to keep backward binary compatibility), and in that case calling memset() on the struct will keep the behavior as it was in the past

     I believe, as a developer, in defensive programming and this is a good practice to handle API which is evolving.

     I hope that I answered your question.

     Thanks
     Dotan

     Reply

   • Yanfang Le says:October 12, 2015

     • Hi Dotan,

       I am trying to flush the work requests in qp, so I first turn the qp state to error state, and then convert to the RTS state. During this process, I don't know where can I get the ah_attr information. Could you tell me how can I get the ah_attr information? Thanks.

       I am trying to copy the information from the old qp (before turn the state to error), however, it doesn't work.

     • Dotan Barak says:October 17, 2015

     • Hi.

       When the QP is in ERROR state,
       you cannot trust the attributes which were queried from it, expect for the QP state.

       If you must keep the AH attributes, maybe you can:
       1) Save a local copy of the attributes as a function of the QP number
       2) Use the qp_context pointer to hold a buffer which holds the AH attributes.

       Thanks
       Dotan

     • Haomai says:January 21, 2015

3. Hi, if I make a connected qp to error state, is it possible that peer side can detect this.

   For example, if we use tcp socket to communicate, if we close one side's socket, other side will get "0" when read. So peer side can know peer socket already closed. Is it possible that rdma can achieve this?

   Thanks!

4. Dotan Barak says:January 23, 2015

   • Hi Haomai.

     In RDMA there aren't any keep-alive messages that send automatically,
     if needed you need to perform this in your application.

     If you close one side, only if you are using rdmacm, the remote side will be informed on this;
     but not in "standard" data verbs.

     If the other side (which didn't move to Error) won't send any message - it won't know that its remote
     QP moved to Error.

     Thanks
     Dotan

     ReplyAdrian says:March 23, 2015

5. Hi Dotan,

   I've just started with verbs and when I try to modify a qp from INIT to RTR state, a 101 error is returned. The 101 error means #define ENETUNREACH 101 /* Network is unreachable */. The values are exchanged via TCP sockets (lid=4 qp_num=116 psn=9105394 from client to server and lid=3 qp_num=116 psn=14618275 from server to client). I don't know what is happening and there is difficult to obtain information.

   Thanks

   ReplyDotan Barak says:March 24, 2015

   • Hi Adrian.

     Are you using explicit ibv_modify_qp(() or do you call librdmacm function?
     Are you using IB or RoCE?

     Thanks
     Dotan

     ReplyAdrian says:March 24, 2015

     • I'm using IB and explicit ibv_modify_qp()Dotan Barak says:March 24, 2015

     • It is weird; since this return value isn't really returned ...
       Which QP transport type are you using?
       Can you please write explicit the values that you set + the used mask in the ibv_modify_qp()?

       Thanks
       DotanAdrian says:March 24, 2015

6. Hi Dotan,
   the code is

   attr.qp_state = IBV_QPS_RTR;
   attr.path_mtu = mtu;
   attr.dest_qp_num = rmsg.qp_num[0];
   attr.rq_psn = rmsg.psn[0];
   attr.ah_attr.dlid = rmsg.lid;
   attr.max_dest_rd_atomic = 1;
   attr.min_rnr_timer = 12;
   attr.ah_attr.is_global = 0;
   attr.ah_attr.sl = 0;
   attr.ah_attr.src_path_bits = 0;

   if (ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |
   IBV_QP_DEST_QPN | IBV_QP_RQ_PSN | IBV_QP_MIN_RNR_TIMER |
   IBV_QP_MAX_DEST_RD_ATOMIC)){
   cerr << "Failed to modify QP 1 to RTR." << endl;
   abort();
   }

   lid, qp_num and psn are sent via TCP sockets.

   Thanks

   ReplyDotan Barak says:March 26, 2015

   • Hi.

     I assume that you are using RC QP.
     What about attr.ah_attr.port_num?

     Thanks
     Dotan

     ReplyAdrian says:March 31, 2015

     • Yes, I use RC QP and the port number is 1. Finally, we reinstall the OFED drivers and it works.

       Thank you for your time.Dotan Barak says:March 31, 2015

     • Sure.

       Maybe there were some inconsistencies between the various libraries..

       I'm glad at the end everything worked for you
       :)
       DotanSangwook says:April 14, 2015

   • Hi Adrian & Dotan. Same error happens in my code too.
     Explicit ibv_modify_qp call to change QP state to RTR fails and returns 101. (My code is based on ibv_rc_pingpong program, an example from libibverb library. And ibv_rc_pingpong also fails in my environment.)
     You said the problem is solved after reinstalling OFED drivers.. Can you share your system environment and OFED version?
     (I installed OFED ver 2.4-1.0.4 on Intel Xeon E5-2670 + MLNX ConnectX-3 machine running Ubuntu 14.04 with kernel 3.13.0-48.)

     ReplySangwook says:April 14, 2015

   • Hi, I wrote a comment to ask for Adrian's system environment. But I figured out the cause of my problem- I had to add "gid-index" option in my program because I'm using RoCE. My program worked fine with that option. Anyway, thanks!

     ReplyDotan Barak says:April 16, 2015

     • I'm glad that everything is fine.

       I'll add this note on RoCE, so people won't fail on it again...

       Thanks
       DotanJesus Camacho says:May 5, 2015

7. Hi Dotan!

   I am running the ib_write_lat microbenchmark with OpenSM.

   I am disabling a link while running them and then OpenSM reconfigures all the network and the benchmark continues working without problems.

   If have reduce the number of retries in the QPs to 1 or 2, then ib_wirte_lat crash in some cases now.

   However, if I change the timeout of the QPs, I have the same behaviour.

   I expected to have less retries if I increase the QP timeout. I understand that every retry is triggered after the QP timeout expires.

   Am i right?

   If not, how is it affecting the QP timeout to the number of retries?

   Thanks for your help!

   Best regards,
   Jesus Camacho

   ReplyDotan Barak says:May 5, 2015

   • Hi Jesus.

     Yes, you are right: every retry is triggered after a QP timeout expires.

     There is an internal clock in the RDMA device, and at least after the timeout expires
     (+ a delta which depends on this clock resolution), there is a retransmission.

     If (in your experience) the link will be stable until the last retransmission,
     ib_write_lat will continue working..

     Thanks
     Dotan
     If until the
     What is "important" is the

     ReplyJesus Camacho says:May 6, 2015

     • Hi Dotan,

       I am replacing a LFT instead of switching on the link back.
       It takes less than 2 ms from the time I send the new block to the LFT and I receive its ACK later. Then the new path should be ready in this short period of time.

       In one of my experiments I am using only one retry and timeout 10 (4194.304 usec).
       Although I spend only 4 ms until the next retransmission, I have to spend up to 0.5 seconds for a successful completion when polling the completion queue (ibv_poll_cq using RC).
       Do you know why this takes all this time?

       Thank you!
       JesusDotan Barak says:May 7, 2015

     • Hi Jesus.

       What does LFT mean? (I don't really familiar with this term)
       Are you performing Automatic Path Migartion?
       What are the values of the timeout/retry_count?

       Thanks
       DotanJesus Camacho says:May 8, 2015

8. Hi Dotan,

   instead of using Automatic Path Migration, I am replacing the forwarding tables (trough OpenSM) in some of the switches to avoid the broken link (I just say the switches to take another port to avoid the link that is down). This takes about 2 milliseconds.

   I am using this values in the ib_write_lat microbenchmark:
   timeout 10
   retry_cnt 1

   This seems to work (I am using a small network). Then I understand that the retransmission is OK in less than timeout+timeout.

   I am measuring the time to send a packet from one node to another and this takes usually several milliseconds.

   However, it takes longer after the fault. And most of the time is spent polling the completion queue -ibv_poll_cq- (up to 0.5 seconds).

   Do you know what could be the reason for this?

   Thanks for your time :-)
   Jesus

   ReplyDotan Barak says:May 8, 2015

   • You are welcome
     :)

     I don't have a solid answer here (from knowledge), but I have an assumption:
     RDMA is very fine tune to handle good flow (since this happen most of the time).

     In case of error flow starts (i.e. retransmission, violation, etc.), maybe this flow isn't fully tuned.

     Another option is that this happening because of other issues
     (for example: is the NUMA node is local or remote? do you use CPU affinity to the running process? etc.)

     I would suggest to use a sniffing tool and try to measure this (when the message is send, when an ack is returned, and when there is a Work Completion available in the CQ) or send an email to the support team of the HW vendor about this.

     BTW, making the SM change the subnet may cause the subnet to start recovery flows
     (since *maybe* client reregister events are created in the subnet's nodes).

     Thanks
     Dotan

     Reply

9. 

   Anonymous says:October 1, 2015

   Hi Dotan,

   When I tried to modify retry_cnt to 7 using ibv_modify_qp,
   I got "Invalid argument". Do you have any suggestion ?
   I listed my code here:

   struct ibv_qp_attr attr;
   int qret;
   attr.retry_cnt = (uint8_t)7;
   attr.timeout = (uint8_t)14;
   attr.rnr_retry = (uint8_t)7;
   qret = ibv_modify_qp(id->qp, &attr, IBV_QP_RETRY_CNT|IBV_QP_TIMEOU T|IBV_QP_RNR_RETRY);

   Thanks

   ReplyDotan Barak says:October 1, 2015

   • Hi.

     From which QP state to which QP state did you try to do this?

     I suspect that you need more flags in the QP transition that you made.

     Thanks
     Dotan

     ReplyJunhyun says:August 12, 2016

10. Hi Dotan, I recently built an event-driven RDMA send/recv IO engine for multiplexing multiple QP events and so far I worked out most kinks, but it suffers when more than one connection is introduced.
    What frustrates me is that the connection works perfectly until I get the WC for my first WR! (QP transition and post_send doesn't return any errors, even with the second connection)
    The second connection keeps throwing me status 12 (RETRY_EXC_ERROR) WCs while the first connection keeps on exchanging data smoothly. When I retry the post_send, I get status 5, vendor_err 249. I took it to mean that the QP has already transitioned to an error state.

    Environment:
    I use dual-port Mellanox CX-3 EN 10GBe on RoCE. Port 1 acts both as the port for my shell connection and RoCE.
    I have made it so that all QPs from the same device share device context and pd, but they use disjoint MRs.
    Here's the code I use for creating and transitioning QPs.
    Every created QPs follow this exact path.
    I wish I could find more explanation as to why this keeps failing.
    Please tell me if I'm doing anything wrong.
    conn.port is the local port I chose for use, and remote_info members are obtained from the other host over TCP connection. (byte-reordered)
    Again, thanks for maintaining such a helpful site.

    struct ibv_qp_init_attr qp_init_attr;
    memset(&qp_init_attr, 0, sizeof(qp_init_attr));
    qp_init_attr.qp_type = IBV_QPT_RC;
    qp_init_attr.sq_sig_all = 1;
    qp_init_attr.send_cq = conn.cq;
    qp_init_attr.recv_cq = conn.cq;
    qp_init_attr.cap.max_send_wr = DEFAULT_CQ_SIZE / 2;
    qp_init_attr.cap.max_recv_wr = DEFAULT_CQ_SIZE / 2;
    qp_init_attr.cap.max_send_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
    qp_init_attr.cap.max_recv_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
    CHECK_PTR(conn.qp = ibv_create_qp(conn.pd, &qp_init_attr), "Failed to create QP for device " << device << ", port " << port << ", error=" << strerror(errno) << ", pd=" < INIT
    struct ibv_qp_attr to_init;
    int flags;
    memset(&to_init, 0, sizeof(to_init));
    to_init.qp_state = IBV_QPS_INIT;
    to_init.port_num = conn.port_num;
    to_init.pkey_index = 0;
    to_init.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE;
    flags = IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_init, flags), "QP transition failed (RESET -> INIT): " <RTR
    struct ibv_qp_attr to_rtr;
    int flags;
    memset(&to_rtr, 0, sizeof(to_rtr));
    to_rtr.qp_state = IBV_QPS_RTR;
    to_rtr.path_mtu = DEFAULT_QP_MTU;
    to_rtr.dest_qp_num = remote_info.qp_num;
    to_rtr.rq_psn = 0;
    to_rtr.max_dest_rd_atomic = 0;
    to_rtr.min_rnr_timer = 12;
    to_rtr.ah_attr.dlid = remote_info.lid;
    to_rtr.ah_attr.sl = 0;
    to_rtr.ah_attr.src_path_bits = 0;
    to_rtr.ah_attr.port_num = conn.port_num;
    // Begin RoCE specific conf (currently universal)
    to_rtr.ah_attr.is_global = 1;
    memcpy(&to_rtr.ah_attr.grh.dgid, remote_info.gid, sizeof(remote_info.gid));
    to_rtr.ah_attr.grh.flow_label = 0;
    to_rtr.ah_attr.grh.hop_limit = 1;
    to_rtr.ah_attr.grh.sgid_index = 0;
    to_rtr.ah_attr.grh.traffic_class = 0;
    flags = IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
    IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rtr, flags), "QP transition failed (INIT -> RTR): " < RTS
    struct ibv_qp_attr to_rts;
    int flags;
    memset(&to_rts, 0, sizeof(to_rts));
    to_rts.qp_state = IBV_QPS_RTS;
    to_rts.timeout = 14;
    to_rts.retry_cnt = 7;
    to_rts.rnr_retry = 7;
    to_rts.sq_psn = 0;
    to_rts.max_rd_atomic = 0;
    flags = IBV_QP_STATE | IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT |
    IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rts, flags), "QP transition failed (RTR -> RTS)" << getQPDescription(local_info));

    ReplyDotan Barak says:August 18, 2016

    • Hi.

      Please make the following:

      1) verify that the DEFAULT_QP_MTU is less than the interface MTU
      (i.e. if the network I/F MTU is 1500, this value should be 1024).
      2) All remote attributes are correct
      3) Make sure that there is a synchronization between both sides
      (to prevent a case where the sender send a message and the receiver isn't in RTR state,
      or in Error state).

      Thanks
      Dotan

      Reply

11. 

    Arvind says:April 10, 2019

    Why does the state have to go to RTR and then to RTS. Can't we just switch from Init to RTS?

    ReplyDotan Barak says:April 11, 2019

    • Hi.

      * When transitioning to RTR - only the receive attributes are filled (the QP can stay at this state, if it will only receive messages)
      * When transitioning to RTS - only the send attributes are filled

      At the end of the day, this is the what the InfiniBand spec specifies...

      Thanks
      Dotan

      Replyqiuhaonan says:April 19, 2019

12. Hello Dotan.
    I have met an interesting problem. At first, I connected two QPs by exchanging the meta info(QPN, GID and etc) and modifying them to RTS. Then exchanging some data on them. Finally, I closed one QP by ibv_destroy_qp and the other QP got errors when sending data to the closed QP, but when I tried to modify the error QP to RTS, it could return to the RTS state and got errors when trying to send data again. The curious thing was why the error QP could be modified to RTS state when the remote target QP had been destroyed. Aren't there some SYN-like packets to be exchanged between the two QPs? Thanks.
    Haonan

    ReplyDotan Barak says:April 19, 2019

    • Hi.

      I have a feeling that you used Reliable Connected QP.
      When you got the error in the QP, you transitioned it back to RTS;
      however, there wasn't any remote side - so you got (again) transport error.

      No, there isn't any "SYN" like packets - they are replaced with Communication Manager
      (but you didn't use the Communication Manager, you replaced it with a socket).

      The behavior is as expected
      :)

      Thanks
      Dotan

      Replyqiuhaonan says:April 19, 2019

13. Hi Dotan.
    Yes, I used RC QP and verbs for this test.
    Thanks.

    ReplyWeizhi Liao says:December 30, 2019

14. Hi Dotan：
    After reading this blog，I feel confused about the required and optional attributes of QP on status transitions 。 the Linux manpages 、rdma_aware_network_programming_user_manual and your blog have different description 。which is right? Thanks

    ReplyDotan Barak says:February 22, 2020

    • Hi.

      What do you mean?
      Can you give me an example?

      Thanks
      Dotan

      Replyraphael says:May 21, 2020

15. Hi Dotan

    I will use an RC queue pair and Write verb from an FPGA device (w/o RDMA network card, 100GbE MAC). For simulation purpose with a workstation and a connectx5 I use python / Scapy to build a RoCEv2 Write_IMM packet from the data and receiver qp_num, rkey, addr, etc. The receiver is also a Connectx-5.

    it works, I get the data and the Immediate value, but only for 8 transfers. The last RC acknowledge from receiver is a RNR Nak.

    Is it possible to configure the Receiver HCA to continuously recive the data ?
    thanks

    ReplyDotan Barak says:July 10, 2020

    • Hi.

      RNR Nak means that the "Receiver Not Ready", this means that there isn't any Receiver Request in that (responder) QP.
      So, I suggest that you'll post more Receive Requests for that side.

      Thanks
      Dotan

      ReplyXiangyu Zhang says:July 13, 2020

16. Hi Dotan:
    These days I encounter an error:after run ibv_post_send to issue an RDMA READ operation, there is nothing happened, this means that the RDMA READ operation don't appears in CQ. After to read "Help, I've posted and Send Request and it wasn't completed with a corresponding Work Completion. What happened?" in "ibv_post_send" page, I decide to try advise in it. The problem is I can't modify qp attribute, the ibv_modity_qp return -22 all the time, below is my code:
    int ret = 0;
    rdma_addrinfo* addrinfo;
    rdma_addrinfo hints = {};
    struct ibv_qp_attr qp_attr;
    hints.ai_port_space = RDMA_PS_TCP;
    struct ibv_qp *client_qp = nullptr;
    if (rdma_getaddrinfo(const_cast(host.c_str()),
    const_cast(port.c_str()), &hints, &addrinfo)) {
    return errors::InvalidArgument(
    strerror(errno), ": ", "cannot connect to rdma://", host, ":", port);
    }

    ibv_qp_init_attr init_attr = {};
    init_attr.qp_type = IBV_QPT_RC;
    init_attr.cap.max_recv_wr = 1;
    init_attr.cap.max_send_wr = 1024;
    init_attr.cap.max_recv_sge = 1;
    init_attr.cap.max_send_sge = 1;

    rdma_cm_id* id;
    if (rdma_create_ep(&id, addrinfo, nullptr, &init_attr)) {
    rdma_freeaddrinfo(addrinfo);
    return errors::Unavailable(strerror(errno), ": ",
    "cannot create endpoint to rdma://", host, ":",
    port);
    }
    rdma_freeaddrinfo(addrinfo);

    client_qp = id->qp;

    // motify qp's attr.
    bzero(&qp_attr, sizeof(qp_attr));
    qp_attr.qp_state = IBV_QPS_RTS;
    qp_attr.timeout = 14;
    qp_attr.retry_cnt = 7;

    ret = ibv_modify_qp(client_qp, &qp_attr,
    IBV_QP_TIMEOUT |
    IBV_QP_STATE |
    IBV_QP_RETRY_CNT);
    if (ret) {
    fprintf(stderr, "failed to motify qp, %s\n", strerror(errno));
    abort();
    }
    if (rdma_connect(id, nullptr)) {
    rdma_destroy_ep(id);
    return errors::Unavailable(strerror(errno), ": ",
    "cannot connect to rdma://", host, ":", port);
    }

    My rdma NIC type is RoCE, could you help me to see why I can't motify qp statue?

    ReplyDotan Barak says:July 15, 2020

    • Hi.

      What is the QP state before you called ibv_modify_qp()?

      Thanks
      Dotan