linux安装软件命令1003无标题,linux系统安装OFED（infiniband）

计算节点安装OFEDnode

scientific linux5.8 x86_64linux

1.查看系统系统:c++

[root@localhost ~]# cat /etc/redhat-release：git

Scientific Linux release 5.8 (Boron)api

[root@localhost ~]# uname -a网络

Linux localhost.localdomain 2.6.18-308.1.1.el5 #1 SMP Thu Mar 8 09:54:23 EST 2012 x86_64 x86_64 x86_64 GNU/Linuxdom

2.查看分区信息ide

[root@localhost ~]# df -h工具

FilesystemSizeUsed Avail Use% Mounted onpost

/dev/sda61.5T776M1.5T1% /

/dev/sda59.7G2.7G6.6G29% /usr

/dev/sda339G177M37G1% /home

/dev/sda1487M18M444M4% /boot

tmpfs16G016G0% /dev/shm

/dev/sr14.1G4.1G0 100% /media

3.infinband硬件信息：Mellanox

[root@localhost ~]# lspci |grep Mellanox

04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

4.解决依赖包：挂在使用本地yum

1.备份分并删除/etc/yum.repo.d/目录下的yum配置文件，

2.新建一个本地源配置文件123.repo，文件内容以下

[root@localhost ~]# cat /etc/yum.repos.d/123.repo

[dvdserver]

name=yumserver

baseurl=file:///media/SL

enabled=1

gpgcheck=0

3固然你必须使用mount挂在光盘到/media

4.解决依赖包

(1)gcc-c++

yum -y install gcc-c++

Loaded plugins: kernel-module

dvdserver| 1.3 kB00:00

dvdserver/primary| 1.0 MB 00:00

dvdserver3725/3725

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package gcc-c++.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: libstdc++-devel = 4.1.2-52.el5 for package: gcc-c++

--> Processing Dependency: gcc = 4.1.2-52.el5 for package: gcc-c++

--> Running transaction check

---> Package gcc.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc

---> Package libstdc++-devel.x86_64 0:4.1.2-52.el5 set to be updated

dvdserver/filelists| 3.4 MB00:01

--> Running transaction check

---> Package glibc-devel.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: glibc-headers = 2.5-81.el5_8.1 for package: glibc-devel

--> Processing Dependency: glibc-headers for package: glibc-devel

--> Running transaction check

---> Package glibc-headers.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers

--> Processing Dependency: kernel-headers for package: glibc-headers

--> Running transaction check

---> Package kernel-headers.x86_64 0:2.6.18-308.1.1.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

gcc-c++x86_644.1.2-52.el5dvdserver3.8 M

Installing for dependencies:

gccx86_644.1.2-52.el5dvdserver5.3 M

glibc-develx86_642.5-81.el5_8.1dvdserver2.4 M

glibc-headersx86_642.5-81.el5_8.1dvdserver597 k

kernel-headersx86_642.6.18-308.1.1.el5dvdserver1.4 M

libstdc++-develx86_644.1.2-52.el5dvdserver2.8 M

Transaction Summary

================================================================================

Install6 Package(s)

Upgrade0 Package(s)

Total download size: 16 M

Downloading Packages:

--------------------------------------------------------------------------------

Total1.5 GB/s |16 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-headers1/6

Installing: glibc-headers2/6

Installing: glibc-devel3/6

Installing: libstdc++-devel4/6

Installing: gcc5/6

Installing: gcc-c++6/6

Installed:

gcc-c++.x86_64 0:4.1.2-52.el5

Dependency Installed:

gcc.x86_64 0:4.1.2-52.el5

glibc-devel.x86_64 0:2.5-81.el5_8.1

glibc-headers.x86_64 0:2.5-81.el5_8.1

kernel-headers.x86_64 0:2.6.18-308.1.1.el5

libstdc++-devel.x86_64 0:4.1.2-52.el5

Complete!

(2)glib

[root@localhost ~]# yum -y install glib

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib.i386 1:1.2.10-20.el5 set to be updated

---> Package glib.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glibi3861:1.2.10-20.el5dvdserver137 k

glibx86_641:1.2.10-20.el5dvdserver144 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 281 k

Downloading Packages:

--------------------------------------------------------------------------------

Total159 MB/s | 281 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib1/2

Installing: glib2/2

Installed:

glib.i386 1:1.2.10-20.el5glib.x86_64 1:1.2.10-20.el5

Complete!

(3) glib-devel

[root@localhost ~]# yum -y install glib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib-devel.i386 1:1.2.10-20.el5 set to be updated

---> Package glib-devel.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glib-develi3861:1.2.10-20.el5dvdserver39 k

glib-develx86_641:1.2.10-20.el5dvdserver39 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 77 k

Downloading Packages:

--------------------------------------------------------------------------------

Total44 MB/s |77 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib-devel1/2

Installing: glib-devel2/2

Installed:

glib-devel.i386 1:1.2.10-20.el5glib-devel.x86_64 1:1.2.10-20.el5

Complete!

[root@localhost ~]# yum -y install kernel-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package kernel-devel.x86_64 0:2.6.18-308.1.1.el5 set to be installed

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

kernel-develx86_642.6.18-308.1.1.el5dvdserver5.7 M

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 5.7 M

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-devel1/1

Installed:

kernel-devel.x86_64 0:2.6.18-308.1.1.el5

Complete!

(4)zilb-devel

[root@localhost ~]# yum -y install zlib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package zlib-devel.i386 0:1.2.3-4.el5 set to be updated

---> Package zlib-devel.x86_64 0:1.2.3-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

zlib-develi3861.2.3-4.el5dvdserver101 k

zlib-develx86_641.2.3-4.el5dvdserver103 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 204 k

Downloading Packages:

--------------------------------------------------------------------------------

Total743 kB/s | 204 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: zlib-devel1/2

Installing: zlib-devel2/2

Installed:

zlib-devel.i386 0:1.2.3-4.el5zlib-devel.x86_64 0:1.2.3-4.el5

Complete!

(5)rpm-build

[root@localhost ~]# yum -y install rpm-build

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package rpm-build.x86_64 0:4.4.2.3-28.el5_8 set to be updated

--> Processing Dependency: xz for package: rpm-build

--> Processing Dependency: elfutils for package: rpm-build

--> Running transaction check

---> Package elfutils.x86_64 0:0.137-3.el5 set to be updated

--> Processing Dependency: elfutils-libs-x86_64 = 0.137-3.el5 for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.130)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.127)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.126)(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1(ELFUTILS_1.0)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1(ELFUTILS_0.122)(64bit) for package: elfutils

--> Processing Dependency: libdw.so.1()(64bit) for package: elfutils

--> Processing Dependency: libasm.so.1()(64bit) for package: elfutils

---> Package xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Processing Dependency: xz-libs = 4.999.9-0.3.beta.20091007git.el5 for package: xz

--> Processing Dependency: liblzma.so.0()(64bit) for package: xz

--> Running transaction check

---> Package elfutils-libs.x86_64 0:0.137-3.el5 set to be updated

---> Package xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

rpm-buildx86_644.4.2.3-28.el5_8dvdserver303 k

Installing for dependencies:

elfutilsx86_640.137-3.el5dvdserver216 k

elfutils-libsx86_640.137-3.el5dvdserver182 k

xzx86_644.999.9-0.3.beta.20091007git.el5dvdserver146 k

xz-libsx86_644.999.9-0.3.beta.20091007git.el5dvdserver94 k

Transaction Summary

================================================================================

Install5 Package(s)

Upgrade0 Package(s)

Total download size: 941 k

Downloading Packages:

--------------------------------------------------------------------------------

Total189 MB/s | 941 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: xz-libs1/5

Installing: xz2/5

Installing: elfutils-libs3/5

Installing: elfutils4/5

Installing: rpm-build5/5

Installed:

rpm-build.x86_64 0:4.4.2.3-28.el5_8

Dependency Installed:

elfutils.x86_64 0:0.137-3.el5

elfutils-libs.x86_64 0:0.137-3.el5

xz.x86_64 0:4.999.9-0.3.beta.20091007git.el5

xz-libs.x86_64 0:4.999.9-0.3.beta.20091007git.el5

Complete!

(6)redhat-rpm-config

[root@localhost ~]# yum -y install redhat-rpm-config

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package redhat-rpm-config.noarch 0:8.0.45-32.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

redhat-rpm-confignoarch8.0.45-32.el5dvdserver55 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 55 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: redhat-rpm-config1/1

Installed:

redhat-rpm-config.noarch 0:8.0.45-32.el5

Complete!

(7)tcl

[root@localhost ~]# yum -y install tcl

Loaded plugins: kernel-module

Setting up Install Process

Package tcl-8.4.13-4.el5.x86_64 already installed and latest version

Resolving Dependencies

--> Running transaction check

---> Package tcl.i386 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcli3868.4.13-4.el5dvdserver956 k

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 956 k

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl1/1

Installed:

tcl.i386 0:8.4.13-4.el5

Complete!

(8)tcl-devel

[root@localhost ~]# yum -y install tcl-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package tcl-devel.i386 0:8.4.13-4.el5 set to be updated

---> Package tcl-devel.x86_64 0:8.4.13-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

tcl-develi3868.4.13-4.el5dvdserver1.0 M

tcl-develx86_648.4.13-4.el5dvdserver1.0 M

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 2.0 M

Downloading Packages:

--------------------------------------------------------------------------------

Total786 MB/s | 2.0 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: tcl-devel1/2

Installing: tcl-devel2/2

Installed:

tcl-devel.i386 0:8.4.13-4.el5tcl-devel.x86_64 0:8.4.13-4.el5

Complete!

(9)libtool

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# yum -y install libtool

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package libtool.x86_64 0:1.5.22-7.el5_4 set to be updated

--> Processing Dependency: autoconf >= 2.50 for package: libtool

--> Processing Dependency: automake >= 1.4 for package: libtool

--> Running transaction check

---> Package autoconf.noarch 0:2.59-12 set to be updated

--> Processing Dependency: imake for package: autoconf

---> Package automake.noarch 0:1.9.6-2.3.el5 set to be updated

--> Running transaction check

---> Package imake.x86_64 0:1.0.2-3 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

libtoolx86_641.5.22-7.el5_4dvdserver667 k

Installing for dependencies:

autoconfnoarch2.59-12dvdserver648 k

automakenoarch1.9.6-2.3.el5dvdserver476 k

imakex86_641.0.2-3dvdserver318 k

Transaction Summary

================================================================================

Install4 Package(s)

Upgrade0 Package(s)

Total download size: 2.1 M

Downloading Packages:

--------------------------------------------------------------------------------

Total681 MB/s | 2.1 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: imake1/4

Installing: autoconf2/4

Installing: automake3/4

Installing: libtool4/4

Installed:

libtool.x86_64 0:1.5.22-7.el5_4

Dependency Installed:

autoconf.noarch 0:2.59-12automake.noarch 0:1.9.6-2.3.el5

imake.x86_64 0:1.0.2-3

Complete!

注意：计算节点到此依赖包安装完成.注意安装前必定要安装上诉依赖包，不然你等着报错包

5.安装驱动

到mellanox官网下载对应操做系统的版本，本案例因为mellanox官网没有scientific linux5.8的相应驱动，只能下载red hat5.8的操做系统使用源代码当前操做系统编译安装

MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz

值得注意的是这个驱动就算在Red Hat 5.8上安装最好系统是英文版，最好/usr目录10GB

(1)解压到指定目录,进入/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# tar -xzvf MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64.tgz -C /usr/src/

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# cd /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/

目录文件以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ls

docsmlnx_add_kernel_support.shrepodatasrc

firmwaremlnxofedinstallRPMSuninstall.sh

(2)标准安装就报错了，以下

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# ./mlnxofedinstall The 2.6.18-308.1.1.el5 kernel is installed, MLNX_OFED does not have drivers available for this kernel.

You can run mlnx_add_kernel_support.sh in order to to generate an MLNX_OFED package with drivers for this kernel.

根据提示添加kernel吧,固然也报错，不详细说了，没办法只有源代码了以下：进入源代码src/

[root@localhost MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64]# cd /usr/src/MLNX_O

FED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/

解压源代码包src/MLNX_OFED_SRC-1.5.3-4.0.42.tgz

解决方法：

(3).解压出来的源代码目录以下，并执行源代码安装了

[root@localhost src]# cd MLNX_OFED_SRC-1.5.3-4.0.42

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ls

BUILD_IDinstall.plLICENSEofed.confRPMSSRPMSuninstall.sh

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ./install.pl执行脚本

OFED Distribution Software Installation Menu

1) View OFED Installation Guide

2) Install OFED Software(安装OFED驱动)

3) Show Installed Software

4) Configure IPoIB(配置Ib网卡)

5) Uninstall OFED Software

Q) Exit

Select Option [1-5]:2

OFED Distribution Software Installation Menu

1) Basic (OFED modules and basic user level libraries)(只是驱动)

2) HPC (OFED modules and libraries, MPI and diagnostic tools)(驱动和MPI工具)

3) All packages (all of Basic, HPC)(管理节点须要opensmd服务，选择个就对了)

4) Customize(自定义，若是你对这个infinband网络结构够了解选它吧)

5) Packages required by VMA (IB and Eth)

6) Packages required by VMA IB

7) Packages required by VMA Eth

Q) Exit

Select Option [1-7]:1

Below is the list of OFED packages that you have chosen

(some may have been added by the installer due to package dependencies):

ofed-scripts

libibverbs

libibverbs-utils

libibverbs-devel

libmthca

libmverbs

libmverbs-devel

libmlx4

libcxgb3

libnes

libipathverbs

librdmacm

librdmacm-utils

mstflint

ofed-docs

mlnx-ofa_kernel

kernel-mft

过程以下

Build ofed-scripts RPM

Runningrpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_prefix /usr' --define '_exec_prefix /usr' --define '_sysconfdir /etc' --define '_usr /usr' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.src.rpm

Install ofed-scripts RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ofed-scripts-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Build libibverbs RPM

Install libibverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-utils-1.1.5mlnx2-1.x86_64.rpm

Install libibverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libibverbs-devel-1.1.5mlnx2-1.x86_64.rpm

Build libmthca RPM

Install libmthca RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmthca-1.0.6mlnx1-0.1.gbe5eef3.x86_64.rpm

Build libmverbs RPM

Install libmverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-0.1.0-3.15.gd28970e.x86_64.rpm

Install libmverbs-devel RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmverbs-devel-0.1.0-3.15.gd28970e.x86_64.rpm

Build libmlx4 RPM

Install libmlx4 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libmlx4-1.0.2mlnx6-1.x86_64.rpm

Build libcxgb3 RPM

Install libcxgb3 RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libcxgb3-1.3.1-1.x86_64.rpm

Build libnes RPM

Install libnes RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libnes-1.1.1mlnx1-1.x86_64.rpm

Build libipathverbs RPM

Install libipathverbs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/libipathverbs-1.2mlnx1-1.x86_64.rpm

Build librdmacm RPM

Install librdmacm RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-1.0.15-1.x86_64.rpm

Install librdmacm-utils RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/librdmacm-utils-1.0.15-1.x86_64.rpm

Build mstflint RPM

Install mstflint RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mstflint-1.4mlnx4-1.21.gd948ddd.x86_64.rpm

Build ofed-docs RPM

Install ofed-docs RPM:

Running rpm -iv/usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/ofed-docs-1.5.3-OFED.1.5.3.4.0.42.x86_64.rpm

Build mlnx-ofa_kernel RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --nodeps --define '_dist .unsupported' --define 'configure_options--with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx4_ib-mod --with-mlx4_vnic-mod --with-nes-mod --with-qib-mod --with-ipoib-mod' --define 'build_kernel_ib 1' --define 'build_kernel_ib_devel 1' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define 'network_dir /etc/sysconfig/network-scripts' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.src.rpm

Install mlnx-ofa_kernel RPM:

Installing mlnx-ofa_kernel RPM

cmd: rpm -ivh--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/mlnx-ofa_kernel-1.5.3-OFED.1.5.3.4.0.42.g3cb72fe.unsupported.x86_64.rpm

Preparing...##################################################

mlnx-ofa_kernel##################################################

Installing kmod-mlnx-ofa_kernel RPM

Preparing...##################################################

kmod-mlnx-ofa_kernel##################################################

Build kernel-mft RPM

Running rpmbuild --rebuild--define '_topdir /var/tmp//OFED_topdir' --define 'KVERSION 2.6.18-308.1.1.el5' --define 'K_SRC /lib/modules/2.6.18-308.1.1.el5/build' --define '_release 2.6.18_308.1.1.el5' --define 'debug_package %{nil}' --define '_prefix /usr' --define '__arch_install_post %{nil}' /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/SRPMS/kernel-mft-2.7.1-7.src.rpm

Install kernel-mft RPM:

Running rpm -iv--nodeps /usr/src/MLNX_OFED_LINUX-1.5.3-4.0.42-rhel5.8-x86_64/src/MLNX_OFED_SRC-1.5.3-4.0.42/RPMS/sl-release-58-4.sl/x86_64/kernel-mft-2.7.1-2.6.18_308.1.1.el5.x86_64.rpm

The default IPoIB interface configuration is based on DHCP.

Note that a special patch for DHCP is required for supporting IPoIB.

The patch is available under docs/dhcp

If you do not have DHCP, you must change this configuration in the following steps.

Do you want to configure ib0? [Y/n]:y输入y配置你的ib卡文件

The current IPoIB configuration for ib0 is:

# Mellanox Technologies MT27500 Family [ConnectX-3]

DEVICE=ib0

BOOTPROTO=dhcp

HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:F4:52:14:03:00:0D:C1:A1

ONBOOT=no

HOTPLUG=no

DHCP_HOSTNAME=localhost.localdomain

Do you want to change this configuration? [y/N]:y须要修改吗，固然y

Enter an IP Adress: 192.168.1.101(IP)

Enter the Netmask: 255.255.255.0(掩码)

Enter the Network: 192.168.1.0(网络)

Enter the Broadcast Adress: 192.168.1.255(广播地址)

Start Device On Boot? [Y/n]:y开机启动(依赖openibd服务)

Selected configuration:

DEVICE=ib0

IPADDR=192.168.1.101

NETMASK=255.255.255.0

NETWORK=192.168.1.0

BROADCAST=192.168.1.255

ONBOOT=yes

Do you want to save the selected configuration? [Y/n]:y保存ib配置

Do you want to configure ib1? [Y/n]:n根据你需求了，我没开

IPoIB interfaces configured successfully

Press any key to continue ...

Device (15b3:1003):

03:00.0 Network controller [0207]: Mellanox Technologies MT27500 Family [ConnectX-3]

Link Width: 8x

PCI Link Speed: Unknown

Installation finished successfully.

Press any key to continue...

OFED Distribution Software Installation Menu

1) View OFED Installation Guide

2) Install OFED Software

3) Show Installed Software

4) Configure IPoIB

5) Uninstall OFED Software

Q) Exit

Select Option [1-5]:q退出

Exiting

驱动基本安装完成了，哈哈完美........

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 3 openibd on

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# chkconfig --level 5 openibd on设置开启启动服务

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# /etc/init.d/openibd restart手动启动服务命令

Unloading HCA driver:[OK]

Loading HCA driver and Access Layer:[OK]

Setting up InfiniBand network interfaces:

Bringing up interface ib0:[OK]

Determining IP information for ib1... failed.

Bringing up interface ib1:[OK]

Setting up service network . . .[done]

查看ib状态卡信息

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# hca_self_test.ofed

---- Performing Adapter Device Self Test ----

Number of CAs Detected ................. 1

PCI Device Check ....................... PASS

Kernel Arch ............................ x86_64

Host Driver Version .................... OFED-internal-1.5.3-4.0.42: 2.6.18-308.1.1.el5

Host Driver RPM Check .................. PASS

Firmware on CA #0 VPI .................. v2.11.550

Firmware Check on CA #0 (VPI) .......... NA

REASON: NO required fw version

Host Driver Initialization ............. PASS

Number of CA Ports Active .............. 1

Port State of Port #1 on CA #0 (VPI)..... UP 4X FDR10 (InfiniBand)看到这个就OK了，是否UP取决与infiniband网络中必需要有一个安装并开启opensmd服务

Port State of Port #2 on CA #0 (VPI)..... DOWN (InfiniBand)

Error Counter Check on CA #0 (VPI)...... PASS

Kernel Syslog Check .................... PASS

Node GUID on CA #0 (VPI) ............... f4:52:14:03:00:0d:c1:a0

------------------ DONE ---------------------

能够测试ping

rtt min/avg/max/mdev = 0.144/0.521/0.899/0.378 ms

[root@localhost MLNX_OFED_SRC-1.5.3-4.0.42]# ping 192.168.1.201

PING 192.168.1.201 (192.168.1.201) 56(84) bytes of data.

64 bytes from 192.168.1.201: icmp_seq=1 ttl=64 time=3.69 ms

64 bytes from 192.168.1.201: icmp_seq=2 ttl=64 time=0.104 ms

64 bytes from 192.168.1.201: icmp_seq=3 ttl=64 time=0.119 ms

--- 192.168.1.201 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2001ms

rtt min/avg/max/mdev = 0.104/1.305/3.693/1.688 ms

算Ok了啦，值得注意的是若是infiniband网络中必需要有一个安装并开启opensmd服务，否者你infiniband卡状态永远down。

那咱们看看管理节点安装了

scientific linux5.8 x86_64

1.查看系统系统:

[root@localhost ~]# cat /etc/redhat-release：

Scientific Linux release 5.8 (Boron)

[root@localhost ~]# uname -a

Linux localhost.localdomain 2.6.18-308.1.1.el5 #1 SMP Thu Mar 8 09:54:23 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

2.查看分区信息

[root@localhost ~]# df -h

FilesystemSizeUsed Avail Use% Mounted on

/dev/sda61.5T776M1.5T1% /

/dev/sda59.7G2.7G6.6G29% /usr

/dev/sda339G177M37G1% /home

/dev/sda1487M18M444M4% /boot

tmpfs16G016G0% /dev/shm

/dev/sr14.1G4.1G0 100% /media

3.infinband硬件信息：Mellanox

[root@localhost ~]# lspci |grep Mellanox

04:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

4.解决依赖包：挂在使用本地yum

1.备份分并删除/etc/yum.repo.d/目录下的yum配置文件，

2.新建一个本地源配置文件123.repo，文件内容以下

[root@localhost ~]# cat /etc/yum.repos.d/123.repo

[dvdserver]

name=yumserver

baseurl=file:///media/SL

enabled=1

gpgcheck=0

3固然你必须使用mount挂在光盘到/media

4.解决依赖包

(1)gcc-c++

yum -y install gcc-c++

Loaded plugins: kernel-module

dvdserver| 1.3 kB00:00

dvdserver/primary| 1.0 MB00:00

dvdserver3725/3725

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package gcc-c++.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: libstdc++-devel = 4.1.2-52.el5 for package: gcc-c++

--> Processing Dependency: gcc = 4.1.2-52.el5 for package: gcc-c++

--> Running transaction check

---> Package gcc.x86_64 0:4.1.2-52.el5 set to be updated

--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc

---> Package libstdc++-devel.x86_64 0:4.1.2-52.el5 set to be updated

dvdserver/filelists| 3.4 MB00:01

--> Running transaction check

---> Package glibc-devel.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: glibc-headers = 2.5-81.el5_8.1 for package: glibc-devel

--> Processing Dependency: glibc-headers for package: glibc-devel

--> Running transaction check

---> Package glibc-headers.x86_64 0:2.5-81.el5_8.1 set to be updated

--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers

--> Processing Dependency: kernel-headers for package: glibc-headers

--> Running transaction check

---> Package kernel-headers.x86_64 0:2.6.18-308.1.1.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

gcc-c++x86_644.1.2-52.el5dvdserver3.8 M

Installing for dependencies:

gccx86_644.1.2-52.el5dvdserver5.3 M

glibc-develx86_642.5-81.el5_8.1dvdserver2.4 M

glibc-headersx86_642.5-81.el5_8.1dvdserver597 k

kernel-headersx86_642.6.18-308.1.1.el5dvdserver1.4 M

libstdc++-develx86_644.1.2-52.el5dvdserver2.8 M

Transaction Summary

================================================================================

Install6 Package(s)

Upgrade0 Package(s)

Total download size: 16 M

Downloading Packages:

--------------------------------------------------------------------------------

Total1.5 GB/s |16 MB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-headers1/6

Installing: glibc-headers2/6

Installing: glibc-devel3/6

Installing: libstdc++-devel4/6

Installing: gcc5/6

Installing: gcc-c++6/6

Installed:

gcc-c++.x86_64 0:4.1.2-52.el5

Dependency Installed:

gcc.x86_64 0:4.1.2-52.el5

glibc-devel.x86_64 0:2.5-81.el5_8.1

glibc-headers.x86_64 0:2.5-81.el5_8.1

kernel-headers.x86_64 0:2.6.18-308.1.1.el5

libstdc++-devel.x86_64 0:4.1.2-52.el5

Complete!

(2)glib

[root@localhost ~]# yum -y install glib

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib.i386 1:1.2.10-20.el5 set to be updated

---> Package glib.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glibi3861:1.2.10-20.el5 dvdserver137 k

glibx86_641:1.2.10-20.el5dvdserver144 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 281 k

Downloading Packages:

--------------------------------------------------------------------------------

Total159 MB/s | 281 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib1/2

Installing: glib2/2

Installed:

glib.i386 1:1.2.10-20.el5glib.x86_64 1:1.2.10-20.el5

Complete!

(3) glib-devel

[root@localhost ~]# yum -y install glib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package glib-devel.i386 1:1.2.10-20.el5 set to be updated

---> Package glib-devel.x86_64 1:1.2.10-20.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

glib-develi3861:1.2.10-20.el5dvdserver39 k

glib-develx86_641:1.2.10-20.el5dvdserver39 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 77 k

Downloading Packages:

--------------------------------------------------------------------------------

Total44 MB/s |77 kB00:00

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: glib-devel1/2

Installing: glib-devel2/2

Installed:

glib-devel.i386 1:1.2.10-20.el5glib-devel.x86_64 1:1.2.10-20.el5

Complete!

[root@localhost ~]# yum -y install kernel-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package kernel-devel.x86_64 0:2.6.18-308.1.1.el5 set to be installed

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

kernel-develx86_642.6.18-308.1.1.el5dvdserver5.7 M

Transaction Summary

================================================================================

Install1 Package(s)

Upgrade0 Package(s)

Total download size: 5.7 M

Downloading Packages:

Running rpm_check_debug

Running Transaction Test

Finished Transaction Test

Transaction Test Succeeded

Running Transaction

Installing: kernel-devel1/1

Installed:

kernel-devel.x86_64 0:2.6.18-308.1.1.el5

Complete!

(4)zilb-devel

[root@localhost ~]# yum -y install zlib-devel

Loaded plugins: kernel-module

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package zlib-devel.i386 0:1.2.3-4.el5 set to be updated

---> Package zlib-devel.x86_64 0:1.2.3-4.el5 set to be updated

--> Finished Dependency Resolution

Beginning Kernel Module Plugin

Finished Kernel Module Plugin

Dependencies Resolved

================================================================================

PackageArchVersionRepositorySize

================================================================================

Installing:

zlib-develi3861.2.3-4.el5dvdserver101 k

zlib-develx86_641.2.3-4.el5dvdserver103 k

Transaction Summary

================================================================================

Install2 Package(s)

Upgrade0 Package(s)

Total download size: 204 k

Downloading Packages:

--------------------------------------------------------------------------------

Total743 kB/s | 204 kB 00:00