linux高性能集群搭建,sge linux高性能集群的搭建与使用
[toc]
sge高性能集群的搭建与使用
集群环境的准备Node1(master)
CentOS7.4
iptables/selinux(off)
IP:10.180.66.11
hostname:node1
ali yum源
Node2(slave)
CentOS7.4
iptables/selinux(off)
IP:10.180.66.12
hostname:node2
ali yum源
Node3(slave)
CentOS7.4
iptables/selinux(off)
IP:10.180.66.13
hostname:node3
ali yum源
Node4(slave)
CentOS7.4
iptables/selinux(off)
IP:10.180.66.14
hostname:node4
ali yum源
Node5(slave)
CentOS7.4
iptables/selinux(off)
IP:10.180.66.15
hostname:node5
ali yum源
master 节点安装
安装相关依赖包1# yum -y install jemalloc-devel openssl-devel ncurses-devel pam-devel libXmu-devel hwloc-devel hwloc hwloc-libs java-devel javacc ant-junit libdb-devel motif-devel csh ksh xterm db4-utils perl-XML-Simple perl-Env xorg-x11-fonts-ISO8859-1-100dpi xorg-x11-fonts-ISO8859-1-75dpi
新建sge管理员用户1
2
3# groupadd -g 490 sgeadmin
# useradd -u 495 -g 490 -r -m -d /home/sgeadmin -s /bin/bash -c "SGE Admin" sgeadmin
# sed -i '/^%wheel/a\%sgeadmin ALL=(ALL) NOPASSWD: ALL' /etc/sudoers
安装sge
sge 链接 密码:c7hy
1
2
3
4
5# cd /usr/local/src/
# tar -xvf ge2011.11.tar.gz
# mkdir -pv /data
# cp -a ge2011.11 /data/sge
# chown sgeadmin.sgeadmin /data/sge
qmaster 安装自动回答脚本，依赖软件包expect, 所有节点都需要安装
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160# cd /data/sge/
# vim master.sh
#!/bin/bash
user="sgeadmin"
/usr/bin/expect <
spawn ./install_qmaster
expect "*>>"
send "
"
expect "*>>"
send "y
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "n
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect eof
EOF
# sh master.sh
修改主节点环境变量
1
2
3
4
5
6# export SGE_ROOT=/data/sge
# echo 'export SGE_ROOT=/data/sge' >> ~/.bashrc
# echo 'PATH=$PATH:/data/sge/bin/linux-x64/:/data/sge/bin/' >> ~/.bashrc
# cp /data/sge/default/common/settings.sh /etc/profile.d/
# sh /etc/profile.d/settings.sh
# source /etc/profile
添加节点
1
2
3
4
5# qconf -ah node1
# qconf -ah node2
# qconf -ah node3
# qconf -ah node4
# qconf -ah node5
master 服务器搭建 nfs 服务
所有节点都需要安装
1# yum -y install nfs-utils
master节点操作
1
2
3# vim /etc/exports
/data/sge 10.180.66.0/24(rw,sync)
# systemctl restart nfs
slave 节点挂载
(node2,node3,node4,node5)执行
1
2
3# mkdir /data/sge -pv
# mount -t nfs 10.180.66.11:/data/sge /data/sge/
# chown sgeadmin.sgeadmin /data/
slave 服务器安装sgeexecd
(node2,node3,node4,node5) 执行
1
2
3# yum -y install hwloc-devel
# useradd -u 495 -g 490 -r -m -d /home/sgeadmin -s /bin/bash -c "SGE Admin" sgeadmin
# sed -i '/^%wheel/a\%sgeadmin ALL=(ALL) NOPASSWD: ALL' /etc/sudoers
生效环境变量
1
2
3
4
5
6# echo 'export SGE_ROOT=/data/sge' >> ~/.bashrc
# echo 'PATH=$PATH:/data/sge/bin/linux-x64/:/data/sge/bin/' >> ~/.bashrc
# echo 'export SGE_CELL=default' >> ~/.bashrc
# cp /data/sge/default/common/settings.sh /etc/profile.d/ -a
# source ~/.bashrc
# source /etc/profile
进行安装，所有节点都执行此脚本
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61# vim slave.sh
# cat slave.sh
#!/bin/bash
user="sgeadmin"
/usr/bin/expect <
spawn ./install_execd
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect "*>>"
send "
"
expect eof
EOF
# sh slave.sh
完成集群搭建1
2
3
4
5
6
7
8
9# qhost
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
node1 linux-x64 1 0.01 968.3M 193.0M 2.0G 64.0K
node2 linux-x64 1 0.01 976.3M 151.0M 2.0G 0.0
node3 linux-x64 1 0.02 978.3M 152.2M 2.0G 84.0K
node4 linux-x64 1 0.02 976.3M 155.4M 2.0G 0.0
node5 linux-x64 1 0.01 978.3M 148.5M 2.0G 84.0K
sge集群的使用
linux高性能集群搭建,sge linux高性能集群的搭建与使用相关推荐
- linux系统中MongoDB数据库安装及分片集群搭建
史上最全的mongodb分片集群搭建,从介绍安装到集群搭建授权,你再也找不到比他更加详细的资料了,未经允许禁止转载!! 一.简介 MongoDB是一个便于开发和扩展设计的文档数据库,属于NoSQL数据 ...
- 使用LVS（Linux Virtual Server）在Linux上搭建负载均衡的集群服务
使用LVS(Linux Virtual Server)在Linux上搭建负载均衡的集群服务 一.基于于NAT的LVS的安装与配置. 1. 硬件需求和网络拓扑 ...
- Linux上搭建Hadoop2.6.3集群以及WIN7通过Eclipse开发MapReduce的demo
随笔 - 70 文章 - 0 评论 - 88 Linux上搭建Hadoop2.6.3集群以及WIN7通过Eclipse开发MapReduce的demo 近期为了分析国内航空旅游业常见安全漏洞,想到 ...
- centos7搭建Linux集群,CentOS 7下Kafka集群安装
一.环境操作系统和软件版本介绍 1.环境操作系统为CentOS Linux release 7.2.1511 (Core) 可用cat /etc/RedHat-release查询 2.软件版本 Kaf ...
- Redis集群环境之linux搭建多机版---已完结，跟着一步一步来你就可以集群成功
上一篇踩着各种坑写了Redis集群环境之linux搭建单机版,这一篇准备就多机版集群进行搭建,主要目的一来是在上一篇的基础上进行精华提粹总结,二来是把单机版与多机版的区别进行记录. 首先软硬件环境: ...
- 高性能网站架构之缓存篇—Redis集群搭建
看过 高性能网站架构之缓存篇--Redis安装配置和高性能网站架构之缓存篇--Redis使用配置端口转发 这两篇文章的,相信你已经对redis有一定的了解,并能够安装上,进行简单的使用了,但是在咱们的 ...
- Linux系统搭建Solr和Redis集群详细步骤
1 搭建Solr集群(SolrCloud) 1.1 Zookeeper介绍 ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoo ...
- Linux环境快速搭建elasticsearch6.5.4集群和Head插件
今天来一起实战Linux环境下搭建elasticsearch6.5.4集群和Head插件: 环境信息 操作系统:Ubuntu 16.04.5 LTS JDK:1.8.0_191 elasticsear ...
- Windows环境下安装HBase(Hadoop3.1.3、HBase2.2.5）与Linux环境下搭建HBase高可用集群
Windows环境下安装HBase 下载HBase 官网: https://hbase.apache.org/downloads.html 不同版本集合:https://archive.apache. ...
- 多台es服务器搭建elasticsearch集群，es单机伪集群搭建，在一台服务器上以三个不同的端口模拟三台主机完成es伪集群搭建，windows和Linux操作方法一致
目录 (一)es单机伪集群搭建 一.准备三台elasticsearch服务器(复制三个es目录) 二.修改每台服务器的配置 1.修改es配置文件 2.配置注意 (二)多台服务器es集群搭建: 1 安装 ...
最新文章
- java ee jsp程序_JavaEE——jsp介绍
- hadoop程序MapReduce之SingletonTableJoin
- SEO交换链接时需遵循哪些原则？
- [Vue 牛刀小试]：第八章 - 组件的基础知识
- LiveVideoStack线上分享第五季（九）：语音合成中的风格控制
- skype linux 安装,Ubuntu 12.04 AMD64 安装 Skype
- 转载 300年前的黑色“巫女”
- codeigniter 操作 图标
- CVPR | BASNet：边缘感知的显著性物体检测
- php 页面拖动改变大小,鼠标拖动改变DIV等网页元素的大小的实现方法
- CTSC2017 APIO2017 THUSC2017 游记
- 渗透中Meterpreter基本操作和对应的windows上的排查或者现象
- Android Button 实现透明 + 圆角按钮效果
- 动态代理(JDK动态代理)初步详解
- Vue3 Extraneous non-props attributes (id) were passed to component but could not be automatically
- speedoffice（Word）怎么添加页码
- APP合集，简单总结一下
- 华为服务器虚拟机登录密码,虚拟机登录密码忘记了怎么办
- 手把手教你脑电波采集及信号处理分析 - 知乎
- STM32电子钟万年历Proteus仿真_LCD1602显示