Linux与云计算——第二阶段Linux服务器架设

第五章：存储Storage服务器架设—分布式存储GlusterFS基础

1 GlusterFS GlusterFS安装

Install GlusterFS to Configure Storage Cluster.

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on all Nodes in Cluster.

[root@node01 ~]# curl http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo -o /etc/yum.repos.d/glusterfs-epel.repo

# enable EPEL, too

[root@node01 ~]# yum --enablerepo=epel -y install glusterfs-server

[root@node01 ~]# systemctl start glusterd

[root@node01 ~]# systemctl enable glusterd

[2] If Firewalld is running, allow GlusterFS service on all nodes.

[root@node01 ~]# firewall-cmd --add-service=glusterfs --permanent

success

[root@node01 ~]# firewall-cmd --reload

Success

It's OK if you mount GlusterFS volumes from clients with GlusterFS Native Client.

[3] GlusterFS supports NFS (v3), so if you mount GlusterFS volumes from clients with NFS, Configure additinally like follows.

[root@node01 ~]# yum -y install rpcbind

[root@node01 ~]# systemctl start rpcbind

[root@node01 ~]# systemctl enable rpcbind

[root@node01 ~]# systemctl restart glusterd

[4] Installing and Basic Settings of GlusterFS are OK. Refer to next section for settings of clustering.

2 设置Distributed

Configure Storage Clustering.

For example, create a distributed volume with 2 servers.

This example shows to use 2 servers but it's possible to use more than 3 servers.

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |

|   node01.srv.world   +----------+----------+    node02.srv.world  |

|                      |                     |                      |

+----------------------+                     +----------------------+

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on All Nodes, refer to here.

[2] Create a Directory for GlusterFS Volume on all Nodes.

[root@node01 ~]# mkdir /glusterfs/distributed

[3] Configure Clustering like follows on a node. (it's OK on any node)

# probe the node

[root@node01 ~]# gluster peer probe node02

peer probe: success.

# show status

[root@node01 ~]# gluster peer status

Number of Peers: 1

Hostname: node02

Uuid: 2ca22769-28a1-4204-9957-886579db2231

State: Peer in Cluster (Connected)

# create volume

[root@node01 ~]# gluster volume create vol_distributed transport tcp \

node01:/glusterfs/distributed \

node02:/glusterfs/distributed

volume create: vol_distributed: success: please start the volume to access data

# start volume

[root@node01 ~]# gluster volume start vol_distributed

volume start: vol_distributed: success

# show volume info

[root@node01 ~]# gluster volume info

Volume Name: vol_distributed

Type: Distribute

Volume ID: 6677caa9-9aab-4c1a-83e5-2921ee78150d

Status: Started

Number of Bricks: 2

Transport-type: tcp

Bricks:

Brick1: node01:/glusterfs/distributed

Brick2: node02:/glusterfs/distributed

Options Reconfigured:

performance.readdir-ahead: on

[4] To mount GlusterFS volume on clients, refer to here.

3 设置Replication

Configure Storage Clustering.

For example, create a Replication volume with 2 servers.

This example shows to use 2 servers but it's possible to use more than 3 servers.

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |

|   node01.srv.world   +----------+----------+   node02.srv.world   |

|                      |                     |                      |

+----------------------+                     +----------------------+

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on All Nodes, refer to here.

[2] Create a Directory for GlusterFS Volume on all Nodes.

[root@node01 ~]# mkdir /glusterfs/replica

[3] Configure Clustering like follows on a node. (it's OK on any node)

# probe the node

[root@node01 ~]# gluster peer probe node02

peer probe: success.

# show status

[root@node01 ~]# gluster peer status

Number of Peers: 1

Hostname: node02

Uuid: 2ca22769-28a1-4204-9957-886579db2231

State: Peer in Cluster (Connected)

# create volume

[root@node01 ~]# gluster volume create vol_replica replica 2 transport tcp \

node01:/glusterfs/replica \

node02:/glusterfs/replica

volume create: vol_replica: success: please start the volume to access data

# start volume

[root@node01 ~]# gluster volume start vol_replica

volume start: vol_replica: success

# show volume info

[root@node01 ~]# gluster volume info

Volume Name: vol_replica

Type: Replicate

Volume ID: 0d5d5ef7-bdfa-416c-8046-205c4d9766e6

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: node01:/glusterfs/replica

Brick2: node02:/glusterfs/replica

Options Reconfigured:

performance.readdir-ahead: on

[4] To mount GlusterFS volume on clients, refer to here.

4 设置Striping

Configure Storage Clustering.

For example, create a Striping volume with 2 servers.

This example shows to use 2 servers but it's possible to use more than 3 servers.

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |

|   node01.srv.world   +----------+----------+   node02.srv.world   |

|                      |                     |                      |

+----------------------+                     +----------------------+

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on All Nodes, refer to here.

[2] Create a Directory for GlusterFS Volume on all Nodes.

[root@node01 ~]# mkdir /glusterfs/striped

[3] Configure Clustering like follows on a node. (it's OK on any node)

# probe the node

[root@node01 ~]# gluster peer probe node02

peer probe: success.

# show status

[root@node01 ~]# gluster peer status

Number of Peers: 1

Hostname: node02

Uuid: 2ca22769-28a1-4204-9957-886579db2231

State: Peer in Cluster (Connected)

# create volume

[root@node01 ~]# gluster volume create vol_striped stripe 2 transport tcp \

node01:/glusterfs/striped \

node02:/glusterfs/striped

volume create: vol_striped: success: please start the volume to access data

# start volume

[root@node01 ~]# gluster volume start vol_striped

volume start: vol_replica: success

# show volume info

[root@node01 ~]# gluster volume info

Volume Name: vol_striped

Type: Stripe

Volume ID: b6f6b090-3856-418c-aed3-bc430db91dc6

Status: Started

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: node01:/glusterfs/striped

Brick2: node02:/glusterfs/striped

Options Reconfigured:

performance.readdir-ahead: on

[4] To mount GlusterFS volume on clients, refer to here.

5 Distributed+Replication

Configure Storage Clustering.

For example, create a Distributed + Replication volume with 4 servers.

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |

|   node01.srv.world   +----------+----------+   node02.srv.world   |

|                      |          |          |                      |

+----------------------+          |          +----------------------+

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#3] |10.0.0.53 | 10.0.0.54| [GlusterFS Server#4] |

|   node03.srv.world   +----------+----------+   node04.srv.world   |

|                      |                     |                      |

+----------------------+                     +----------------------+

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on All Nodes, refer to here.

[2] Create a Directory for GlusterFS Volume on all Nodes.

[root@node01 ~]# mkdir /glusterfs/dist-replica

[3] Configure Clustering like follows on a node. (it's OK on any node)

# probe the node

[root@node01 ~]# gluster peer probe node02

peer probe: success.

[root@node01 ~]# gluster peer probe node03

peer probe: success.

[root@node01 ~]# gluster peer probe node04

peer probe: success.

# show status

[root@node01 ~]# gluster peer status

Number of Peers: 3

Hostname: node02

Uuid: 2ca22769-28a1-4204-9957-886579db2231

State: Peer in Cluster (Connected)

Hostname: node03

Uuid: 79cff591-1e98-4617-953c-0d3e334cf96a

State: Peer in Cluster (Connected)

Hostname: node04

Uuid: 779ab1b3-fda9-46da-af95-ba56477bf638

State: Peer in Cluster (Connected)

# create volume

[root@node01 ~]# gluster volume create vol_dist-replica replica 2 transport tcp \

node01:/glusterfs/dist-replica \

node02:/glusterfs/dist-replica \

node03:/glusterfs/dist-replica \

node04:/glusterfs/dist-replica

volume create: vol_dist-replica: success: please start the volume to access data

# start volume

[root@node01 ~]# gluster volume start vol_dist-replica

volume start: vol_dist-replica: success

# show volume info

[root@node01 ~]# gluster volume info

Volume Name: vol_dist-replica

Type: Distributed-Replicate

Volume ID: 784d2953-6599-4102-afc2-9069932894cc

Status: Started

Number of Bricks: 2 x 2 = 4

Transport-type: tcp

Bricks:

Brick1: node01:/glusterfs/dist-replica

Brick2: node02:/glusterfs/dist-replica

Brick3: node03:/glusterfs/dist-replica

Brick4: node04:/glusterfs/dist-replica

Options Reconfigured:

performance.readdir-ahead: on

[4] To mount GlusterFS volume on clients, refer to here.

6 Striping+Replication

Configure Storage Clustering.

For example, create a Striping + Replication volume with 4 servers.

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#1] |10.0.0.51 | 10.0.0.52| [GlusterFS Server#2] |

|   node01.srv.world   +----------+----------+   node02.srv.world   |

|                      |          |          |                      |

+----------------------+          |          +----------------------+

|

+----------------------+          |          +----------------------+

| [GlusterFS Server#3] |10.0.0.53 | 10.0.0.54| [GlusterFS Server#4] |

|   node03.srv.world   +----------+----------+   node04.srv.world   |

|                      |                     |                      |

+----------------------+                     +----------------------+

It is recommended to use partitions for GlusterFS volumes which are different from the / partition.

The environment on this example is set as that sdb1 is mounted on /glusterfs directory for GlusterFS Configuration on all Nodes.

[1] Install GlusterFS Server on All Nodes, refer to here.

[2] Create a Directory for GlusterFS Volume on all Nodes.

[root@node01 ~]# mkdir /glusterfs/strip-replica

[3] Configure Clustering like follows on a node. (it's OK on any node)

# probe the node

[root@node01 ~]# gluster peer probe node02

peer probe: success.

[root@node01 ~]# gluster peer probe node03

peer probe: success.

[root@node01 ~]# gluster peer probe node04

peer probe: success.

# show status

[root@node01 ~]# gluster peer status

Number of Peers: 3

Hostname: node02

Uuid: 2ca22769-28a1-4204-9957-886579db2231

State: Peer in Cluster (Connected)

Hostname: node03

Uuid: 79cff591-1e98-4617-953c-0d3e334cf96a

State: Peer in Cluster (Connected)

Hostname: node04

Uuid: 779ab1b3-fda9-46da-af95-ba56477bf638

State: Peer in Cluster (Connected)

# create volume

[root@node01 ~]# gluster volume create vol_strip-replica stripe 2 replica 2 transport tcp \

node01:/glusterfs/strip-replica \

node02:/glusterfs/strip-replica \

node03:/glusterfs/strip-replica \

node04:/glusterfs/strip-replica

volume create: vol_strip-replica: success: please start the volume to access data

# start volume

[root@node01 ~]# gluster volume start vol_strip-replica

volume start: vol_strip-replica: success

# show volume info

[root@node01 ~]# gluster volume info

Volume Name: vol_strip-replica

Type: Striped-Replicate

Volume ID: ec36b0d3-8467-47f6-aa83-1020555f58b6

Status: Started

Number of Bricks: 1 x 2 x 2 = 4

Transport-type: tcp

Bricks:

Brick1: node01:/glusterfs/strip-replica

Brick2: node02:/glusterfs/strip-replica

Brick3: node03:/glusterfs/strip-replica

Brick4: node04:/glusterfs/strip-replica

Options Reconfigured:

performance.readdir-ahead: on

[4] To mount GlusterFS volume on clients, refer to here.

7 Clients Settings

It's the settings for GlusterFS clients to mount GlusterFS volumes.

[1] For mounting with GlusterFS Native Client, Configure like follows.

[root@client ~]# curl http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo -o /etc/yum.repos.d/glusterfs-epel.repo

[root@client ~]# yum -y install glusterfs glusterfs-fuse

# mount vol_distributed volume on /mnt

[root@client ~]# mount -t glusterfs node01.srv.world:/vol_distributed /mnt

[root@client ~]# df -hT

Filesystem                           Type            Size  Used Avail Use% Mounted on

/dev/mapper/centos-root              xfs              27G  1.1G   26G   5% /

devtmpfs                             devtmpfs        2.0G     0  2.0G   0% /dev

tmpfs                                tmpfs           2.0G     0  2.0G   0% /dev/shm

tmpfs                                tmpfs           2.0G  8.3M  2.0G   1% /run

tmpfs                                tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup

/dev/vda1                            xfs             497M  151M  347M  31% /boot

node01.srv.world:/vol_distributed fuse.glusterfs   40G   65M   40G   1% /mnt

[2] NFS (v3) is also supported, so it's possible to mount with NFS.

Configure for it on GlusterFS Servers first, refer to here.

[root@client ~]# yum -y install nfs-utils

[root@client ~]# systemctl start rpcbind rpc-statd

[root@client ~]# systemctl enable rpcbind rpc-statd

[root@client ~]# mount -t nfs -o mountvers=3 node01.srv.world:/vol_distributed /mnt

[root@client ~]# df -hT

Filesystem                           Type      Size  Used Avail Use% Mounted on

/dev/mapper/centos-root              xfs        27G  1.1G   26G   5% /

devtmpfs                             devtmpfs  2.0G     0  2.0G   0% /dev

tmpfs                                tmpfs     2.0G     0  2.0G   0% /dev/shm

tmpfs                                tmpfs     2.0G  8.3M  2.0G   1% /run

tmpfs                                tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup

/dev/vda1                            xfs       497M  151M  347M  31% /boot

node01.srv.world:/vol_distributed nfs        40G   64M   40G   1% /mnt

---

详细视频课程请戳—→ http://edu.51cto.com/course/course_id-6574.html

---