author：headsen chen

date： 2019-01-18  10:22:20

notice：created  by headsen chen himself and not allowed to copy, or you will count law question!

版本环境：centos6.8 ，64位，内核：2.6.32

1，配置网卡：

在新卡装上机器，接收光纤，两根线都有接，而且是反接的方式接，接通后，光纤灯会亮

2，安装软件RDMA的方式安装，编译内核和用户态，重启进入新内核4.7

3，安装驱动：

正常的kernel安装方法(2.6的内核)

# /mnt/mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64/install

这里必须采用这种，因为是新内核4.7

#tar fx mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64.tgz#cd mlnx-en-4.4-2.0.7.0-rhel6.8-x86_64#./install --add-kernel-support --skip-repo

Logs dir:/tmp/mlnx-en.28728.logs

General log file:/tmp/mlnx-en.28728.logs/general.log

Verifying KMP rpms compatibility with target kernel...

This program will install the mlnx-en package on your machine.

Note that all other Mellanox, OEM, OFED, RDMAorDistribution IB packages will be removed.

Those packages are removed due to conflicts with mlnx-en, do notreinstall them.

Do you want tocontinue?[y/N]:y

rpm--nosignature -e --allmatches --nodeps rdma rdma

Starting mlnx-en-4.4-2.0.7.0installation ...

Installing mlnx-en-utils 4.4RPM

Preparing...##################################################

mlnx-en-utils ##################################################

Installing kmod-mlnx-en 4.4RPM

Preparing...##################################################

kmod-mlnx-en ##################################################

Installing mlnx-en-sources 4.4RPM

Preparing...##################################################

mlnx-en-sources ##################################################

Installing mlnx-en-doc 4.4RPM

Preparing...##################################################

mlnx-en-doc ##################################################

Installing user level RPMs:

Preparing...##################################################

ofed-scripts ##################################################

Preparing... ##################################################

mstflint ##################################################

Device (83:00.0):83:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4Lx]

Link Width: x8

PCI Link Speed: 8GT/s

Device (83:00.1):83:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4Lx]

Link Width: x8

PCI Link Speed: 8GT/s

Installation finished successfully.

Preparing...########################################### [100%]

1:mlnx-fw-updater ########################################### [100%]

Updated /usr/share/hwdata/pci.ids

Attempting to perform Firmware update...

Querying Mellanox devices firmware ...

Device#1:

----------Device Type: ConnectX4LX

Part Number: MCX4121A-XCA_Ax

Description: ConnectX-4 Lx EN network interface card; 10GbE dual-port SFP28; PCIe3.0 x8; ROHS R6

PSID: MT_2420110004

PCI Device Name:83:00.0Base MAC: ec0d9ad2fd68

Versions: Current Available

FW14.20.1010 14.23.1020PXE3.5.0210 3.5.0504UEFI N/A 14.16.0017Status: Update required---------Found1device(s) requiring firmware update...

Device#1: Updating FW ... Done

Restart neededforupdates to take effect.

Log File:/tmp/mlnx-en.28728.logs/fw_update.log

Configuring/etc/security/limits.conf.

To load the new driver, run:/etc/init.d/mlnx-en.d restart

4，重启服务：

/etc/init.d/mlnx-en.d restart

5，安装MLNX_OFED_LINUX-4.4

这里不用像软件RDMA 那样的启动rxe_cfg了。

yum -y install libmml tcl tk libmnl

tar fx MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64.tgz

cd MLNX_OFED_LINUX-4.4-2.0.7.0-rhel6.8-x86_64

./mlnxofedinstall --add-kernel-support --skip-repo/etc/init.d/openibd restart #这个命令最好在管理卡上执行，xshell上执行有可能导致网卡掉IP，

/etc/init.d/network restart

chkconfig openibd on

ibv_devices

出现一下结果代表成功：#ibv_devices

device node GUID------ ----------------mlx5_1 ec0d9a0300d2fc99

mlx5_0 ec0d9a0300d2fc98

如果这一步不成功(有时候rxe_cfg不启动也可以)：#rxe_cfg start (并绑定eth4网卡)

#ibv_devices

device node GUID------ ----------------rxe0 ee0d9afffed2fd68

#ibv_devinfo rxe0

hca_id: rxe0

transport: InfiniBand (0)

fw_ver:0.0.0

node_guid: ee0d:9aff:fed2:fd68

sys_image_guid:0000:0000:0000:0000vendor_id:0x0000vendor_part_id: 0

hw_ver:0x0phys_port_cnt:1port:1state: PORT_ACTIVE (4)

max_mtu:4096 (5)

active_mtu:1024 (3)

sm_lid: 0

port_lid: 0

port_lmc:0x00link_layer: Ethernet

6，利用rping 命令测试：

生成server端：

[[email protected]:/root]#rping -s -a 10.20.15.23 -v -C 10

生成client端：

client端的安装和服务端一样，生成命令是：

#rping -c -a 10.20.15.23 -v -C 10

此时就会出现一下界面，证明安装成功：

#rping -s -a 10.20.15.23 -v -C 10

server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr

server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs

server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu

server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv

server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw

server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx

server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy

server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz

server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA

server DISCONNECT EVENT...

waitfor RDMA_READ_ADV state 10

#rping -c -a 10.20.15.23 -v -C 10

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr

ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs

ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu

ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv

ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw

ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx

ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy

ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz

ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA

client DISCONNECT EVENT...

------------------------------------------------------

利用udaddy来测试，出现一下结果代表成功：

x86_64]#udaddy

udaddy: starting server

receiving data transfers

sending replies

data transfers complete

test completereturn status 0

root]#udaddy -s 10.20.15.23

udaddy: starting client

udaddy: connecting

initiating data transfers

receiving data transfers

data transfers complete

test completereturn status 0

原文：https://www.cnblogs.com/kaishirenshi/p/10286270.html