及配置准备

安装

中放在/soft_ins下，当然也可以建一个 NFS 服务器共享安装。

[NSD1][root][/home/scripts/gpfs]>rcp_file.sh /home/scripts/gpfs/install_gpfs.sh

[NSD1][root][/home/scripts/gpfs]> run_cmd.sh /home/scripts/gpfs/install_gpfs.shITPUB个人空间r]+zC|3Q

确认每台服务器安装显示 ok

[NSD1][root][/home/scripts/gpfs]>run_cmd.sh /home/scripts/gpfs/chang_profile.shITPUB个人空间 F1@ @ R:N

[NSD1][root][/home/scripts/gpfs]>run_cmd.sh mkdir /share

[NSD1][root][/home/scripts/gpfs]>run_cmd.sh ln -s /share /tmp/mmfs

[NSD1][root][/home/scripts/gpfs]>run_cmd.sh mkdir /tmp/gpfsITPUB个人空间_$D_6bd

ITPUB个人空间0L$ke\8b(I&zmH

ITPUB个人空间:`6j\-Lk7B7Y i.v

r6mzF"G[0

)ARe ve'y#S+B/[ {0

Y8]t6\BS)zX'q8G0ITPUB个人空间4ll"m^+L6Wz

[NSD1][root][/]>vi /tmp/gpfs/nodefile

NSD1:quorum

NSD2:quorum

App1:client

App2:client

App3:client

App4:client

2q$I8h c Q:SU b0

先确保清理干净原先 GPFS 集群

[NSD1][root][/tmp/gpfs]>mmdelnode -f

2o6sGvH]F&r0

建立集群

[NSD1][root][/tmp/gpfs]>mmcrcluster -C bgbcrun -U bgbc

-N /tmp/gpfs/nodefile -p NSD1 -s NSD2

Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD1

Thu Jun 28 15:42:57 BEIST 2007: 6027-1664 mmcrcluster: Processing node NSD2

…..

mmcrcluster: Command successfully completed

mmcrcluster: 6027-1371 Propagating the cluster configuration to all

affected nodes. This is an asynchronous process.

其中参数含义

-C bgbcrun 设定集群名称

-U bgbc 定义域名

-N /tmp/gpfs/nodefile 指定节点文件名

-p NSD1 指定主NSD服务器为 NSD1

-s NSD1 指定备NSD服务器为 NSD1

Cn-^ e0y dk0

[NSD1][root][/tmp/gpfs]>mmlscluster

GPFS cluster information

========================

GPFS cluster name: bgbcrun.NSD1

GPFS cluster id: 739157013761844865

GPFS UID domain: bgbc

Remote shell command: /usr/bin/rsh

Remote file copy command: /usr/bin/rcp

GPFS cluster configuration servers:

-----------------------------------

Primary : NSD1

Secondary server: NSD2

Node Daemon node name IP address Admin node name Designation|-------10--------20--------30--------40--------50--------60--------70--------80--------9||-------- XML error: The previous line is longer than the max of 90 characters ---------|-----------------------------------------------------------------------------------------

1 NSD1 10.66.3.98 NSD1 quorum

2 NSD2 10.66.3.99 NSD2 quorum

3 App1 10.66.5.51 App1

4 App2 10.66.5.52 App2

5 App3 10.66.5.53 App3

6 App4 10.66.5.54 App4ITPUB个人空间7?N0R$@IDH

W'\i]lz s)I0

)Ks?7K `XWK0

ITPUB个人空间Q-h\}/|7LG8X

ITPUB个人空间S7E;N#Jou9Oq

ITPUB个人空间/P'P? u[(o!HITPUB个人空间hMh+sx\

[NSD1][root][/tmp/gpfs]>vi /tmp/gpfs/nsdfile

添加

hdisk2:NSD1:NSD2: dataAndMetadata:4ITPUB个人空间A'vuOu

[NSD1][root][/tmp/gpfs]>mmcrnsd -F /tmp/gpfs/nsdfile

mmcrnsd: Processing disk hdisk2

mmcrnsd: 6027-1371 Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.ITPUB个人空间K'jA0t*k#}A

此时，该文件作了转换

[NSD1][root][/tmp/gpfs]>cat nsdfile

# hdisk2:NSD2:NSD1: dataAndMetadata:4

gpfs1nsd:::dataAndMetadata:4:

ZZ,w4B$kYbVd0

[NSD1][root][/tmp/gpfs]>lspv

hdisk3 00003e846ffa7a6e gpfs1nsd

tU3V(H'MsOU} M4r0

[NSD2][root][/tmp/gpfs]>mmstartup -a

Thu Jun 28 15:52:12 BEIST 2007: 6027-1642 mmstartup: Starting GPFS ...

NSD2: 6027-2114 The GPFS subsystem is already active.

…

App4: 6027-2114 The GPFS subsystem is already active.

j8Auy5Yo#f0ITPUB个人空间X:Hc7\a/DXbR^W2Z

JQ7jz n wgg`0

ITPUB个人空间{/b7E[c;Qfk@hN(Jl

ITPUB个人空间#C!h'O|0f2z1g_ S

$oK)Dg.fM0ITPUB个人空间 ^MST*kn

[NSD2][root][/]>mmcrfs /share sharelv -F /tmp/gpfs/nsdfile -A yes -B 64K -n 30 -v no

GPFS: 6027-531 The following disks of sharelv will be formatted on node NSD1:

gpfs1nsd: size 67108864 KB

GPFS: 6027-540 Formatting file system ...

GPFS: 6027-535 Disks up to size 140 GB can be added to storage pool 'system'.

Creating Inode File

Creating Allocation Maps

Clearing Inode Allocation Map

Clearing Block Allocation Map

GPFS: 6027-572 Completed creation of file system /dev/sharelv.

mmcrfs: 6027-1371 Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

参数含义如下：

/share 文件系统 mount 点名

sharelv 指定文件系统 lv 名

-F 指定 NSD 的文件名

-A 自动 mount 选项为 yes

-B 块大小为64K

-n 挂载文件系统的节点估计数30个

-v 校验建立磁盘是否已有文件系统 为否ITPUB个人空间|jp*O"S?1P6`@

[NSD2][root][/home/scripts/gpfs]>run_cmd.sh mount /shareITPUB个人空间&FMO"r(gI%i

[App1][root][/]>mkdir /share/user1work

[App1][root][/]>chown user1:bea /share/user1work

[App1][root][/]>chmod 700 /share/user1work

[App1][root][/]>mkdir /share/user1temp

[App1][root][/]>chown user1:bea /share/user1temp

[App1][root][/]> chmod 750 /share/user1tempITPUB个人空间iTe c h i

同样在另3台机器建立其他目录并修改相应所有者即权限，即 work 目录只有所有者可操作，temp 目录可读可进入。ITPUB个人空间 P\#F*d9Uy+g:G

ITPUB个人空间0jMSGY,sA5M'`

ITPUB个人空间]]ZgU@4P9j

ITPUB个人空间vg{S1nH0g9kb7t_J

5h3a6zrTRc0

在节点开机后自动启动 GPFS

[NSD1][root][/]>mmchconfig autoload=yes

mmchconfig: Command successfully completed

mmchconfig: 6027-1371 Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.

;X,}G9[#F,ha.Sy?3Y0

采用多个 Quorum 盘

[NSD1][root][/]> mmchconfig singleNodeQuorum=no

mmchconfig: 6027-1119 Obsolete option: singleNodeQuorum

"^e u!b/Jz7JR0

[NSD1][root][/]>mmlsconfig

Configuration data for cluster bgbcrun.NSD2:

-----------------------------------------------

clusterName bgbcrun.NSD2

clusterId 739157013761844865

clusterType lc

autoload no

useDiskLease yes

uidDomain bgbc

maxFeatureLevelAllowed 912

/W"Iz_4X0

至此 GPFS 的安装配置完成

[NSD1][root][/]>mmgetstate -a

Node number Node name GPFS state

------------------------------------------

1 NSD1 active

2 NSD2 active

3 App1 active

4 App2 active

5 App3 active

10 App4 activeITPUB个人空间MG].u'oN.E\)AQ

Sd]Y9LRA.g(s1w3{0

] n\#M0mYg![0

ITPUB个人空间({&C8~y(G)of"Y

C`Y d:_Z0

/fmO8J.n-v0

最终，通过 userid 的不同，在 App1 的共享目录 /share下，user1 只能读写自己文件夹的文件，对其它用户的 temp 目录可读，对其它用户的 work 目录不能读写。

[App1][/share]>ls -latr

drwx------ 2 user1 bea 2048 Jun 28 17:21 user1work

drwxr--x-- 2 user1 bea 2048 Jun 28 17:21 user1temp

drwx------ 2 502 bea 2048 Jun 28 17:22 user2work

drwxr--x--- 2 502 bea 2048 Jun 28 17:22 user2temp

drwx------ 2 503 bea 2048 Jun 28 17:22 user3work

drwxr-x-- 2 503 bea 2048 Jun 28 17:22 user3temp

drwxr---- 2 504 bea 2048 Jun 28 17:22 user4work

drwxr-x--- 2 504 bea 2048 Jun 28 17:23 user4temp

[App1][user1][/share]>cd user2work

ksh: user2temp : Permission denied.

[App1][user1][/share]>cd user2temp

[App1][user1][/share]>>a

The file access permissions do not allow the specified action.

ksh[2]: a: 0403-005 Cannot create the specified file.

[App1][user1][/share]>cd /user1temp

[App1][user1][/share/user1temp]>>a

[App1][user1][/share/user1temp]>ls -l

-rw-r----- 1 user1 bea 0 Aug 13 18:47 a

[App1][user1][/share/xhtemp]>rm a

[App1][user1][/share/xhtemp]>ls -l

[App1][/share]>ls -latr

drwxr------ 2 user1 bea 2048 Jun 28 17:21 user1work

drwxr------ 2 user1 bea 2048 Jun 28 17:21 user1temp

drwxr------ 2 502 bea 2048 Jun 28 17:22 user2work

drwxr------ 2 502 bea 2048 Jun 28 17:22 user2temp

drwxr------ 2 503 bea 2048 Jun 28 17:22 user3work

drwxr------ 2 503 bea 2048 Jun 28 17:22 user3temp

drwxr------ 2 504 bea 2048 Jun 28 17:22 user4work

drwxr------ 2 504 bea 2048 Jun 28 17:23 user4temp

-r}l#b-W5b)m0

同样，在其它3台机器也有类似我们希望的结果。

!P(RD6T*A+O g:c5[+^ }0

[ y)n.{zX2h1f0

5v {N ` V q0

(H!V*v mB:O;\K,N7w0

1.新增一个节点

除所有准备和安装工作外，还需完成以下操作：

[NSD1][root][/home/scripts/gpfs]>mmaddnode -N bgbcw14:client

Thu Jun 28 16:28:21 BEIST 2007: 6027-1664 mmaddnode: Processing node App3

mmaddnode: Command successfully completed

mmaddnode: 6027-1371 Propagating the cluster configuration data to all

affected nodes. This is an asynchronous process.ITPUB个人空间i&W]1p}M tQ

2.停掉节点

停掉所有节点

[NSD2][root][/home/scripts/gpfs]>mmshutdown -a

Mon Jul 30 09:56:02 BEIST 2007: 6027-1341 mmshutdown: Starting force unmount of

GPFS file systems

NSD1: forced unmount of /share

…

App4: forced unmount of /share

Mon Jul 30 09:56:07 BEIST 2007: 6027-1344 mmshutdown: Shutting down GPFS daemons

NSD1: Shutting down!

…

App3: Shutting down!

NSD1: 'shutdown' command about to kill process 368890

….

App4: 'shutdown' command about to kill process 474040

Mon Jul 30 09:56:13 BEIST 2007: 6027-1345 mmshutdown: Finished

4h1l \B6L P0v"yk g#J4G0

也可以用 mmshutdown -N 只停某个节点

清除 GPFS

1.fuser –kcu /share

2.unmount /share #在所有节点

3.mmdelfs sharelv,

4.mmlsfs sharelv #检查结果

5.mmdelnsd –F /tmp/gpfs/nsdfile

6.mmshutdown –a

7.mmdelnode –n /tmp/gpfs/nodefile

8.mdelnode –f #最后清除集群