heartbeat+DRBD+mysql高可用集群实战

四台主机

主机名                 IP地址             用途

dbm128                               172.16.0.128                      heartbeat+DRBD+MYSQL的primary节点

dbm129                               172.16.0.129                      heartbeat+DRBD+mysql 的secondary节点

dbs130                                 172.16.0.130                      slave(mater_host=172.16.0.233)

dbs131                                 172.16.0.131                      slave(mater_host=172.16.0.233)

配置前的准备

1.设置hostname及解析

编辑/etc/hosts文件，添加如下内容：

172.16.0.128   dbm128

172.16.0.129   dbm129

2.新增加一块/dev/sdb5硬盘存放DRBD数据（两台都增加）

命令如下：

[root@dbm128 ~]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel. Changes will remain in memory only,

until you decide to write them. After that, of course, the previous

content won't be recoverable.

The number of cylinders for this disk is set to 6527.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n

Command action

e   extended

p   primary partition (1-4)

e

Partition number (1-4): 1

First cylinder (1-6527, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-6527, default 6527):

Using default value 6527

Command (m for help): n

Command action

l   logical (5 or over)

p   primary partition (1-4)

l

First cylinder (1-6527, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-6527, default 6527):

Using default value 6527

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

3.DRBD的部署（两台机器都这么做）

yum -y install gcc  gcc-c++  kernel*  安装好后重启一下，免得后面出错

#wget http://oss.linbit.com/drbd/8.4/drbd-8.4.3.tar.gz

# tar zxvf drbd-8.4.3.tar.gz

#cd drbd-8.4.3

#./configure --prefix=/ --with-km

#make

#make install

4. 加载安装DRDB模块

#modprobe drbd

机器重启后DRBD模块消失，需要写入/etc/rc.local开机自动执行

通过lsmod来检查是否已经执行成功

[root@dbm128 drbd-8.4.3]# lsmod | grep drbd

drbd                  327968  0

5.编辑配置文件（两台配置一样dbm128，dbm129）

如果后面有问题要把LINUX防火墙关闭，或者把端口增加过去

[root@dbm128 drbd-8.4.3]# cat /etc/drbd.conf

global {usage-count yes;}

common{syncer{rate 200M;}}

resource r0{

protocol C;

net{

cram-hmac-alg sha1;

shared-secret "FooFunFactory";

}

disk{

on-io-error detach;

fencing resource-only;

}

startup{

wfc-timeout 120;

degr-wfc-timeout 120;

}

device /dev/drbd1;

on dbm128{

disk /dev/sdb5;

address 172.16.0.128:7789;

meta-disk internal;

}

on dbm129{

disk /dev/sdb5;

address 172.16.0.129:7789;

meta-disk internal;

}

}

 

6.启动DRBD(两台操作)

可通过如下命令启动DRBD

[root@dbm128 drbd-8.4.3]# drbdadm create-md all

[root@dbm128 drbd-8.4.3]# /etc/init.d/drbd start

[root@dbm128 drbd-8.4.3]# drbdadm -- --overwrite-data-of-peer primary all （这一步只在主服务器上做）

在主机上查看DRBD状态

[root@dbm128 ~]# cat /proc/drbd

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm128, 2014-07-18 23:01:15

1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r---n-

ns:45789492 nr:0 dw:0 dr:45794304 al:0 bm:2794 lo:0 pe:3 ua:5 ap:0 ep:1 wo:f oos:6639292

[================>...] sync'ed: 87.4% (6480/51196)M

finish: 0:06:46 speed: 16,336 (18,732) K/sec

以上数据正在同步。下面同步完成

[root@dbm128 ~]# cat /proc/drbd

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm128, 2014-07-18 23:01:15

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----

ns:52426428 nr:0 dw:0 dr:52426428 al:0 bm:3200 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

在从机上查看DRBD状态

[root@dbm129 ~]# cat /proc/drbd

version: 8.4.3 (api:1/proto:86-101)

GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@dbm129, 2014-07-18 23:14:34

1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----

ns:0 nr:52426428 dw:52426428 dr:0 al:0 bm:3200 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

在执行过程中，如果出现上述信息，代表操作成功

7.格式化DRBD文件系统（只在主机上做）

[root@dbm128 ~]# mkfs.ext3 /dev/drbd1

注意开机不要自启动DRBD服务，因为要通过heartbeat来启动DRBD

8.挂载DRBD分区

[root@dbm128 ~]# mkdir /data

[root@dbm128 ~]# mount /dev/drbd1 /data

[root@dbm128 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

92G  4.9G   82G   6% /

/dev/sda1              99M   35M   59M  38% /boot

tmpfs                 501M     0  501M   0% /dev/shm

/dev/drbd1             50G  180M   47G   1% /data

后面就把mysql数据目录放在/data上

9.安装MYSQL数据库（四台都要安装）

安装所需要系统库相关库文件

[root@dbm128 ~]#yum -y install gcc gcc-c++ gcc-g77 autoconf automake zlib* fiex* libxml* ncurses-devel libmcrypt* libtool-ltdl-devel*  cmake

[root@dbm128 ~]# groupadd mysql

[root@dbm128 ~]# useradd -r -g mysql mysql

[root@dbm128 ~]# tar zxvf mysql-5.5.25.tar.gz

[root@dbm128 ~]# cd mysql-5.5.25

[root@dbm128 mysql-5.5.25]#  cmake -DCMAKE_INSTALL_PREFIX=/opt/mysql/ -DMYSQL_DATADIR=/opt/mysql/data -DMYSQL_UNIX_ADDR=/opt/mysql/data/mysqld.sock -DWITH_INNOBASE_STORAGE_ENGINE=1  -DENABLED_LOCAL_INFILE=1 -DMYSQL_TCP_PORT=3306 -DEXTRA_CHARSETS=all -DDEFAULT_CHARSET=utf8 -DDEFAULT_COLLATION=utf8_general_ci -DMYSQL_UNIX_ADDR=/opt/mysql/data/mysql.sock -DMYSQL_USER=mysql  -DWITH_DEBUG=0

[root@dbm128 mysql-5.5.25]# make && make install

赋予相关的权限

[root@dbm128 mysql]# chown -R mysql:mysql /opt/mysql

[root@dbm128 mysql]#  /opt/mysql/scripts/mysql_install_db --user=mysql --basedir=/opt/mysql   --datadir=/opt/mysql/data

mysql配置文件

[root@dbm128 mysql]# cp /opt/mysql/support-files/mysql.server /etc/init.d/mysql

[root@dbm128 mysql]# cp /opt/mysql/support-files/my-large.cnf /etc/my.cnf

[root@dbm128 mysql]# chmod +x /etc/init.d/mysql

dbm128上操作就可以，因为会同步过去所以dbm129不用操作

[root@dbm128 mysql]# cp -r /opt/mysql/data /data/mysql

[root@dbm128 mysql]# chown -R mysql:mysql /data/mysql

[root@dbm128 mysql]# vi /etc/init.d/mysql   (两台要修改dbm128,dbm129)

找到datadir=/opt/mysql/data 修改成datadir=/data/mysql

[root@dbm128 mysql]#/etc/init.d/mysql start 启动（dbm129不用去启动到时做heartbeat采用自动启动）

[root@dbm128 mysql]# /opt/mysql/bin/mysqladmin -u root password '123456' （三台都操作，dbm129不用操作已经同步过去了）

10.安装heartbeat(两台机器都安装)

要先安装DRBD,然后才能安装hearbeat

安装命令如下：

[root@dbm128 ~]# yum -y install heartbeat heartbeat-devel  heartbeat-stonith heartbeat-pils

接下来配置文件

[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/ha.cf /etc/ha.d/ha.cf

[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/haresources /etc/ha.d/haresources

[root@dbm128 ~]# cp /usr/share/doc/heartbeat-2.1.3/authkeys /etc/ha.d/authkeys

[root@dbm128 ~]# chmod 600 /etc/ha.d/authkeys

修改ha.cf

[root@dbm128 ~]# cat /etc/ha.d/ha.cf

debugfile /var/log/ha-debug

logfile       /var/log/ha-log

logfacility local0

keepalive 1

deadtime 15

warntime 5

initdead 30

ucast eth0 172.16.0.129 (修改对方的节点IP)

ping_group group 172.16.0.128 172.16.0.129

auto_failback on (从节点修改为off)

node dbm128

node dbm129

respawn hacluster /usr/lib64/heartbeat/ipfail

respawn hacluster /usr/lib64/heartbeat/dopd

apiauth ipfail gid=haclient uid=hacluster

apiauth dopd gid=haclient uid=hacluster

增加到后面cat  /etc/ha.d/authkeys

auth 1

1 sha1 HA_JACKBILLOW

增加到后面 cat /etc/ha.d/haresources

dbm128  drbddisk::r0 Filesystem::/dev/drbd1::/data mysql  IPaddr::172.16.0.233/24/eth0

上面dbm128主机，如果dbm129修改本主机名。

启动heartbeat服务：

[root@dbm128 ~]/etc/init.d/heartbeat start

[root@dbm129 ~]/etc/init.d/heartbeat start

配置heartbeat 在启动级别3中自动启动:

[root@dbm128 ~]chkconfig --level 35 heartbeat on

[root@dbm129 ~]chkconfig --level 35 heartbeat on

观察heartbeat 启动日志是否正常

[root@dbm128 ~]# tail -f /var/log/ha-log

IPaddr[4782]:  2014/07/21_16:48:15 INFO:  Success

heartbeat[4147]: 2014/07/21_16:48:57 info: dbm129 wants to go standby [foreign]

heartbeat[4147]: 2014/07/21_16:48:57 info: standby: acquire [foreign] resources from dbm129

heartbeat[5029]: 2014/07/21_16:48:57 info: acquire local HA resources (standby).

ResourceManager[5042]:        2014/07/21_16:48:57 info: Acquiring resource group: dbm128 drbddisk::r0 Filesystem::/dev/drbd1::/data mysql IPaddr::172.16.0.233/24/eth0

Filesystem[5083]:    2014/07/21_16:48:58 INFO:  Running OK

IPaddr[5158]:  2014/07/21_16:48:58 INFO:  Running OK

heartbeat[5029]: 2014/07/21_16:48:58 info: local HA resource acquisition completed (standby).

heartbeat[4147]: 2014/07/21_16:48:58 info: Standby resource acquisition done [foreign].

heartbeat[4147]: 2014/07/21_16:48:58 info: remote resource transition completed.

测试一下，看172.16.0.233是VIP，在两台主机之间漂移。

11.主从复制的配置

修改dbs130,dbs131主的my.cnf的server-id=1，每个同步服务都必须设定一个唯一的编号。

在Master（这里为dbm128机器）上增加一个用于复制的账号：

mysql>GRANT REPLICATION SLAVE ON *.*  TO 'repl'@'172.16.0.%'  IDENTIFIED BY '123456';

mysql>FLUSH REPLICATION;

mysql> show master status;

+------------------+----------+--------------+------------------+

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |

+------------------+----------+--------------+------------------+

| mysql-bin.000018 |      107 |              |                  |

+------------------+----------+--------------+------------------+

1 row in set (0.00 sec)

设置Slave 主机

修改my.cnf的server-id,内容如下：

dbs130主机

server-id=3

dbs130主机

server-id=4

开启Master与Slave的同步

在Slave上执行如下命令

mysql>CHANGE MASTER TO MASTER_HOST='172.16.0.233',

->MASTER_USER='repl',

->MASTER_PASSWORD='123456',

->MASTER_LOG_FILE='mysql-bin.000018',

->MASTER_LOG_POS=107;

之后执行

mysql>slave start;

mysql> show slave status\G

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 172.16.0.233

Master_User: repl

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.000018

Read_Master_Log_Pos: 107

Relay_Log_File: dbs130-relay-bin.000007

Relay_Log_Pos: 253

Relay_Master_Log_File: mysql-bin.000018

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 107

Relay_Log_Space: 556

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

Replicate_Ignore_Server_Ids:

Master_Server_Id: 1

1 row in set (0.00 sec)

在其他几台slave机器上也执行上面的命令