ProxySQL+MGR实现读写分离和主节点故障无感知切换
2024-05-10 00:07:27
ProxySQL+MGR实现读写分离和主节点故障无感知切换
一、环境准备
192.168.153.149 mgr-node1
192.168.153.150 mgr-node2
192.168.153.151 mgr-node3
192.168.153.152 proxysql-node
实验前关闭防火墙和SELinux
二、MGR(组复制)部署过程
1、四台机器互相做本地解析
[root@proxysql-node ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.153.149 mgr-node1
192.168.153.150 mgr-node2
192.168.153.151 mgr-node3
192.168.153.152 proxysql-node
2、四台机器安装扩展源
[root@proxysql-node yum.repos.d]# yum -y install epel-release.noarch
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile* base: mirrors.cn99.com* extras: mirrors.huaweicloud.com* updates: mirrors.cn99.com
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution
3、三台数据库节点下载MySQL数据库
1、MySQL yum包下载(三台机器)
[root@mgr-node1 ~]# wget http://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
--2020-07-18 14:19:46-- http://repo.mysql.com/mysql57-community-release-el7-10.noarch.rpm
Resolving repo.mysql.com (repo.mysql.com)... 23.36.53.20
Connecting to repo.mysql.com (repo.mysql.com)|23.36.53.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25548 (25K) [application/x-redhat-package-manager]
Saving to: ‘mysql57-community-release-el7-10.noarch.rpm’100%[====================================================================================>] 25,548 132KB/s in 0.2s2020-07-18 14:19:50 (132 KB/s) - ‘mysql57-community-release-el7-10.noarch.rpm’ saved [25548/25548]2、MySQL 软件源安装（三台机器）
[root@mgr-node1 ~]# yum -y install mysql57-community-release-el7-10.noarch.rpm
Loaded plugins: fastestmirror
Examining mysql57-community-release-el7-10.noarch.rpm: mysql57-community-release-el7-10.noarch
Marking mysql57-community-release-el7-10.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package mysql57-community-release.noarch 0:el7-10 will be installed
--> Finished Dependency ResolutionDependencies Resolved==============================================================================================================================Package Arch Version Repository Size
==============================================================================================================================
Installing:mysql57-community-release noarch el7-10 /mysql57-community-release-el7-10.noarch 30 kTransaction Summary
==============================================================================================================================
Install 1 PackageTotal size: 30 k
Installed size: 30 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transactionInstalling : mysql57-community-release-el7-10.noarch 1/1Verifying : mysql57-community-release-el7-10.noarch 1/1Installed:mysql57-community-release.noarch 0:el7-10Complete!
3、MySQL 服务安装(三台机器)
[root@mgr-node1 ~]# yum install -y mysql-community-server mysql
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 6.0 kB 00:00:00* base: mirrors.ustc.edu.cn* epel: mirrors.tuna.tsinghua.edu.cn* extras: mirrors.163.com* updates: mirrors.163.com4、启动mysqld 服务（三台机器）
[root@mgr-node1 ~]# systemctl start mysqld
[root@mgr-node1 ~]# systemctl status mysqld.service
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since Sat 2020-07-18 14:39:30 CST; 2min 5s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlProcess: 8617 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)Process: 8567 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)Main PID: 8620 (mysqld)CGroup: /system.slice/mysqld.service└─8620 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pidJul 18 14:39:26 mgr-node1 systemd[1]: Starting MySQL Server...
Jul 18 14:39:30 mgr-node1 systemd[1]: Started MySQL Server.
5、设置登陆MySQL数据库的密码（三台机器）
由于MySQL从5.7开始不允许首次安装后使用空密码进行登录！为了加强安全性，系统会随机生成一个密码以供管理员首次登录使用，这个密码记录在/var/log/mysqld.log文件中，使用下面的命令可以查看此密码：
[root@mgr-node1 ~]# grep 'temporary password' /var/log/mysqld.log
2020-07-18T06:39:27.392196Z 1 [Note] A temporary password is generated for root@localhost: gKizw,a8!gAU
登陆数据库
[root@mgr-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.31Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>
设置登录密码
温馨提示
mysql5.7通过上面默认安装后，执行语句可能会报错：
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
这个报错与Mysql 密码安全策略validate_password_policy的值有关，validate_password_policy可以取0、1、2三个值：
解决办法：
set global validate_password_policy=0;
set global validate_password_length=1;
mysql> set global validate_password_policy=0; #修改密码策略
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_length=1; #修改密码长度
Query OK, 0 rows affected (0.00 sec)mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123.com'; #设置root用户的登陆密码
Query OK, 0 rows affected (0.00 sec)mysql> FLUSH PRIVILEGES; #刷新
Query OK, 0 rows affected (0.00 sec)先给这组MGR起个组名，组名可以随便起，但是不能使用主机的GTID!
通过节点的uuid作为loose-group_replication_group_name的组名，并且每个节点的这个组名必须一样！
这里使用MGR-node1节点mysql里的uuid作为组名
mysql> select uuid();
+--------------------------------------+
| uuid() |
+--------------------------------------+
| 44d4d9a6-c8c6-11ea-9bf7-000c2939d519 |
+--------------------------------------+
4、mgr-node1操作
1、修改my.cnf配置文件
[root@mgr-node1 etc]# cp /etc/my.cnf /etc/my.cnf.bak
[root@mgr-node1 etc]# > my.cnf
[root@mgr-node1 etc]# cat my.cnf
[mysqld]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.socksymbolic-links = 0log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid#GTID:
server_id = 1
gtid_mode = on
enforce_gtid_consistency = onmaster_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE#binlog
log_bin = mysql-bin
log-slave-updates = 1
binlog_format = row
sync-master-info = 1
sync_binlog = 1#relay log
skip_slave_start = 1transaction_write_set_extraction=XXHASH64 #以便在server收集写集合的同时将其记录到二进制日志。写集合基于每行的主键，并且是行更改后的唯一标识此标识将用于检测冲突。
loose-group_replication_group_name="5db40c3c-180c-11e9-afbf-005056ac6820" #组的名字可以随便起,但不能用主机的GTID! 所有节点的这个组名必须保持一致！
loose-group_replication_start_on_boot=off #为了避免每次启动自动引导具有相同名称的第二个组,所以设置为OFF。
loose-group_replication_local_address= "192.168.153.149:24901"
loose-group_replication_group_seeds= "192.168.153.149:24901,192.168.153.150:24901,192.168.153.151:24901"
loose-group_replication_bootstrap_group=off
loose-group_replication_single_primary_mode=off #关闭单主模式的参数（本例测试时多主模式，所以关闭该项）
loose-group_replication_enforce_update_everywhere_checks=on #开启多主模式的参数
loose-group_replication_ip_whitelist="192.168.153.0/24,127.0.0.1/8" # 允许加入组复制的客户机来源的ip白名单
2、重启数据库
[root@mgr-node1 etc]# systemctl restart mysqld
[root@mgr-node1 etc]# systemctl status mysqld
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since Sat 2020-07-18 15:19:19 CST; 4s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlProcess: 8733 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)Process: 8715 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)Main PID: 8736 (mysqld)CGroup: /system.slice/mysqld.service└─8736 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pidJul 18 15:19:18 mgr-node1 systemd[1]: Stopped MySQL Server.
Jul 18 15:19:19 mgr-node1 systemd[1]: Starting MySQL Server...
Jul 18 15:19:19 mgr-node1 systemd[1]: Started MySQL Server.
3、登录数据库操作
[root@mgr-node1 etc]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> SET SQL_LOG_BIN=0; #即不记录二进制日志
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)mysql> GRANT REPLICATION SLAVE ON *.* TO rpl_slave@'%' IDENTIFIED BY '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)mysql> reset master;
Query OK, 0 rows affected (0.00 sec)mysql> SET SQL_LOG_BIN=1;
Query OK, 0 rows affected (0.00 sec)mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='123.com' FOR CHANNEL 'group_replication_recovery';
Query OK, 0 rows affected, 2 warnings (0.35 sec)mysql> INSTALL PLUGIN group_replication SONAME 'group_replication.so'; #安装复制组插件
Query OK, 0 rows affected (0.03 sec)mysql> SHOW PLUGINS;
+----------------------------+----------+--------------------+----------------------+---------+
| Name | Status | Type | Library | License |
+----------------------------+----------+--------------------+----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| sha256_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
| MEMORY | ACTIVE | STORAGE ENGINE | NULL | GPL |
| InnoDB | ACTIVE | STORAGE ENGINE | NULL | GPL |
| INNODB_TRX | ACTIVE | INFORMATION SCHEMA | NULL | GPL |
.
.
.
.
| ngram | ACTIVE | FTPARSER | NULL | GPL |
| validate_password | ACTIVE | VALIDATE PASSWORD | validate_password.so | GPL |
| group_replication | ACTIVE | GROUP REPLICATION | group_replication.so | GPL |
+----------------------------+----------+--------------------+----------------------+---------+
46 rows in set (0.00 sec)mysql> SET GLOBAL group_replication_bootstrap_group=ON;
Query OK, 0 rows affected (0.00 sec)mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.65 sec)mysql> SET GLOBAL group_replication_bootstrap_group=OFF;
Query OK, 0 rows affected (0.00 sec)mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
1 row in set (0.00 sec)比如要保证上面的group_replication_applier的状态为"ONLINE"才对！创建一个测试库
mysql> CREATE DATABASE kevin CHARACTER SET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)mysql> use kevin;
Database changedmysql> create table if not exists haha (id int(10) PRIMARY KEY AUTO_INCREMENT,name varchar(50) NOT NULL);
Query OK, 0 rows affected (0.10 sec)mysql> insert into kevin.haha values(1,"wangshibo"),(2,"guohuihui"),(3,"yangyang"),(4,"shikui");
Query OK, 4 rows affected (0.16 sec)
Records: 4 Duplicates: 0 Warnings: 0mysql> select * from kevin.haha;
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
+----+-----------+
4 rows in set (0.00 sec)5、mgr-node2操作
1、修改my.cnf配置文件
[root@mgr-node2 ~]# cd /etc/
[root@mgr-node2 etc]# cp my.cnf my.cnf.bak
[root@mgr-node2 etc]# > my.cnf
[root@mgr-node2 etc]# cat my.cnf
[mysqld]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.socksymbolic-links = 0log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid#GTID:
server_id = 2
gtid_mode = on
enforce_gtid_consistency = onmaster_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE#binlog
log_bin = mysql-bin
log-slave-updates = 1
binlog_format = row
sync-master-info = 1
sync_binlog = 1#relay log
skip_slave_start = 1transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="5db40c3c-180c-11e9-afbf-005056ac6820"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "192.168.153.150:24901"
loose-group_replication_group_seeds= "192.168.153.149::24901,192.168.153.150::24901,192.168.153.151::24901"
loose-group_replication_bootstrap_group=off
loose-group_replication_single_primary_mode=off
loose-group_replication_enforce_update_everywhere_checks=on
loose-group_replication_ip_whitelist="192.168.153.0/24,127.0.0.1/8"
2、重启数据库
[root@mgr-node2 etc]# systemctl restart mysqld
[root@mgr-node2 etc]# systemctl status mysqld
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since Sat 2020-07-18 15:53:17 CST; 7s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlProcess: 23456 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)Process: 23438 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)Main PID: 23459 (mysqld)CGroup: /system.slice/mysqld.service└─23459 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pidJul 18 15:53:16 mgr-node2 systemd[1]: Stopped MySQL Server.
Jul 18 15:53:16 mgr-node2 systemd[1]: Starting MySQL Server...
Jul 18 15:53:17 mgr-node2 systemd[1]: Started MySQL Server.
3、登录数据库进行操作
[root@mgr-node2 etc]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> SET SQL_LOG_BIN=0;
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)mysql> GRANT REPLICATION SLAVE ON *.* TO rpl_slave@'%' IDENTIFIED BY '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)mysql> reset master;
Query OK, 0 rows affected (0.00 sec)mysql> SET SQL_LOG_BIN=1;
Query OK, 0 rows affected (0.00 sec)mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='123.com' FOR CHANNEL 'group_replication_recovery';
Query OK, 0 rows affected, 2 warnings (0.02 sec)mysql> INSTALL PLUGIN group_replication SONAME 'group_replication.so';
Query OK, 0 rows affected (0.02 sec)mysql> SHOW PLUGINS;
+----------------------------+----------+--------------------+----------------------+---------+
| Name | Status | Type | Library | License |
+----------------------------+----------+--------------------+----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| sha256_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
.
.
.
| BLACKHOLE | ACTIVE | STORAGE ENGINE | NULL | GPL |
| FEDERATED | DISABLED | STORAGE ENGINE | NULL | GPL |
| partition | ACTIVE | STORAGE ENGINE | NULL | GPL |
| ngram | ACTIVE | FTPARSER | NULL | GPL |
| validate_password | ACTIVE | VALIDATE PASSWORD | validate_password.so | GPL |
| group_replication | ACTIVE | GROUP REPLICATION | group_replication.so | GPL |
+----------------------------+----------+--------------------+----------------------+---------+
46 rows in set (0.00 sec)这里只需要执行这一步即可！mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (5.72 sec)查看组内情况，发现MGR-node2已经成功加入这个组内了。
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
| group_replication_applier | 6de78dac-c8c5-11ea-a1ae-000c299642d5 | mgr-node2 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
2 rows in set (0.00 sec)查看下，发现已经将MGR-node1节点添加的数据同步过来了
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| kevin |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)mysql> select * from kevin.haha;
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
+----+-----------+
4 rows in set (0.00 sec)6、mgr-node3操作
1、修改my.cnf 配置文件
[root@mgr-node3 ~]# cd /etc/
[root@mgr-node3 etc]# cp my.cnf my.cnf.bak
[root@mgr-node3 etc]# > my.cnf
[root@mgr-node3 etc]# cat my.cnf
[mysqld]
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.socksymbolic-links = 0log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid#GTID:
server_id = 3
gtid_mode = on
enforce_gtid_consistency = onmaster_info_repository=TABLE
relay_log_info_repository=TABLE
binlog_checksum=NONE#binlog
log_bin = mysql-bin
log-slave-updates = 1
binlog_format = row
sync-master-info = 1
sync_binlog = 1#relay log
skip_slave_start = 1transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="5db40c3c-180c-11e9-afbf-005056ac6820"
loose-group_replication_start_on_boot=off
loose-group_replication_local_address= "192.168.153.151:24901"
loose-group_replication_group_seeds= "192.168.153.149:24901,192.168.153.150:24901,192.168.153.151:24901"
loose-group_replication_bootstrap_group=off
loose-group_replication_single_primary_mode=off
loose-group_replication_enforce_update_everywhere_checks=on
loose-group_replication_ip_whitelist="192.168.153.0/24,127.0.0.1/8"
2、重启数据库
[root@mgr-node3 etc]# systemctl restart mysqld
[root@mgr-node3 etc]# systemctl status mysqld
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: active (running) since Sat 2020-07-18 16:35:02 CST; 7s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlProcess: 23420 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)Process: 23402 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)Main PID: 23423 (mysqld)CGroup: /system.slice/mysqld.service└─23423 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pidJul 18 16:35:01 mgr-node3 systemd[1]: Starting MySQL Server...
Jul 18 16:35:02 mgr-node3 systemd[1]: Started MySQL Server.
3、查看mysql日志
[root@mgr-node3 etc]# grep password /var/log/mysqld.log
2020-07-18T07:10:43.712239Z 1 [Note] A temporary password is generated for root@localhost: 8C>hj(M9oqY9
2020-07-18T07:11:41.418934Z 2 [Warning] Plugin validate_password reported: 'Effective value of validate_password_length is changed. New value is 4'
2020-07-18T08:34:59.882470Z 0 [Note] Shutting down plugin 'validate_password'
2020-07-18T08:35:01.520836Z 0 [Note] Shutting down plugin 'sha256_password'
2020-07-18T08:35:01.520838Z 0 [Note] Shutting down plugin 'mysql_native_password'
4、登录数据库操作
[root@mgr-node3 etc]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)mysql> GRANT REPLICATION SLAVE ON *.* TO rpl_slave@'%' IDENTIFIED BY '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)mysql> reset master;
Query OK, 0 rows affected (0.01 sec)mysql> SET SQL_LOG_BIN=1;
Query OK, 0 rows affected (0.00 sec)mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='123.com' FOR CHANNEL 'group_replication_recovery';
Query OK, 0 rows affected, 2 warnings (0.01 sec)mysql> INSTALL PLUGIN group_replication SONAME 'group_replication.so';
Query OK, 0 rows affected (0.03 sec)mysql> SHOW PLUGINS;
+----------------------------+----------+--------------------+----------------------+---------+
| Name | Status | Type | Library | License |
+----------------------------+----------+--------------------+----------------------+---------+
| binlog | ACTIVE | STORAGE ENGINE | NULL | GPL |
| mysql_native_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| sha256_password | ACTIVE | AUTHENTICATION | NULL | GPL |
| CSV | ACTIVE | STORAGE ENGINE | NULL | GPL |
.
.
.
| validate_password | ACTIVE | VALIDATE PASSWORD | validate_password.so | GPL |
| group_replication | ACTIVE | GROUP REPLICATION | group_replication.so | GPL |
+----------------------------+----------+--------------------+----------------------+---------+
46 rows in set (0.00 sec)这里只需要执行这一步即可！
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (3.75 sec)查看组内情况，发现MGR-node3已经成功加入这个组内了。
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
| group_replication_applier | 6de78dac-c8c5-11ea-a1ae-000c299642d5 | mgr-node2 | 3306 | ONLINE |
| group_replication_applier | cb32290b-c8c5-11ea-a447-000c296697cf | mgr-node3 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)查看下，发现已经将在其他节点上添加的数据同步过来了
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| kevin |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)mysql> select * from kevin.haha;
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
+----+-----------+
4 rows in set (0.00 sec)
7、组复制数据同步测试
1、在任意一个节点上执行
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
| group_replication_applier | 6de78dac-c8c5-11ea-a1ae-000c299642d5 | mgr-node2 | 3306 | ONLINE |
| group_replication_applier | cb32290b-c8c5-11ea-a447-000c296697cf | mgr-node3 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)mysql> insert into kevin.haha values(11,"beijing"),(12,"shanghai"),(13,"anhui");
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0mysql> select * from kevin.haha;
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+-----------+
7 rows in set (0.00 sec)2、在另一台机器上查看数据库
mysql> select * from kevin.haha;
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+-----------+
7 rows in set (0.00 sec)
8、组复制故障测试
1、关闭一台机器的mysql数据库
[root@mgr-node3 etc]# systemctl stop mysqld
[root@mgr-node3 etc]# systemctl status mysqld
● mysqld.service - MySQL ServerLoaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)Active: inactive (dead) since Sat 2020-07-18 16:53:58 CST; 2s agoDocs: man:mysqld(8)http://dev.mysql.com/doc/refman/en/using-systemd.htmlProcess: 23420 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)Process: 23402 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)Main PID: 23423 (code=exited, status=0/SUCCESS)Jul 18 16:35:01 mgr-node3 systemd[1]: Starting MySQL Server...
Jul 18 16:35:02 mgr-node3 systemd[1]: Started MySQL Server.
Jul 18 16:53:46 mgr-node3 systemd[1]: Stopping MySQL Server...
Jul 18 16:53:58 mgr-node3 systemd[1]: Stopped MySQL Server.2、在另一台机器数据库里面查看组的状态
mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
| group_replication_applier | 6de78dac-c8c5-11ea-a1ae-000c299642d5 | mgr-node2 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
2 rows in set (0.00 sec)已经自动把down掉的节点剔除组
如果把down掉的节点重启后 不会自动添加到组里面，需要手动添加.其他节点更新的数据也会及时同步过来！
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (3.23 sec)mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 6ceb6753-c8c1-11ea-bb69-000c2939d519 | mgr-node1 | 3306 | ONLINE |
| group_replication_applier | 6de78dac-c8c5-11ea-a1ae-000c299642d5 | mgr-node2 | 3306 | ONLINE |
| group_replication_applier | cb32290b-c8c5-11ea-a447-000c296697cf | mgr-node3 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
9、三个节点都出现故障解决方法
要是三个节点都发生故障的话，在节点的故障都恢复后，需要手动重新做组复制，操作流程如下：第一个节点
mysql> reset master;
mysql> SET SQL_LOG_BIN=1;
mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='slave@123' FOR CHANNEL 'group_replication_recovery';
mysql> STOP GROUP_REPLICATION;
mysql> SET GLOBAL group_replication_bootstrap_group=ON;
mysql> START GROUP_REPLICATION;
mysql> SET GLOBAL group_replication_bootstrap_group=OFF;
mysql> SELECT * FROM performance_schema.replication_group_members;第二个节点
mysql> reset master;
mysql> SET SQL_LOG_BIN=1;
mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='slave@123' FOR CHANNEL 'group_replication_recovery';
mysql> START GROUP_REPLICATION;
mysql> SELECT * FROM performance_schema.replication_group_members;第三个节点
mysql> reset master;
mysql> SET SQL_LOG_BIN=1;
mysql> CHANGE MASTER TO MASTER_USER='rpl_slave', MASTER_PASSWORD='slave@123' FOR CHANNEL 'group_replication_recovery';
mysql> START GROUP_REPLICATION;
mysql> SELECT * FROM performance_schema.replication_group_members;
三、ProxySQL+MGR 配置过程
1、下载mariadb客户端
[root@proxysql-node ~]# yum install -y mariadb.x86_64 mariadb-libs.x86_64
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 6.0 kB 00:00:00* base: mirrors.huaweicloud.com* epel: mirrors.yun-idc.com* extras: mirrors.huaweicloud.com* updates: mirror.bit.edu.cn2、安装ProxySQL
1、安装
上传包
[root@proxysql-node ~]# ls
anaconda-ks.cfg proxysql-1.4.8-1-centos7.x86_64.rpm[root@proxysql-node ~]# yum install -y perl-DBI perl-DBD-MySQL
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile* base: mirrors.huaweicloud.com* epel: mirrors.yun-idc.com* extras: mirrors.huaweicloud.com* updates: mirror.bit.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package perl-DBD-MySQL.x86_64 0:4.023-6.el7 will be installed
---> Package perl-DBI.x86_64 0:1.627-4.el7 will be installed[root@proxysql-node ~]# rpm -ivh proxysql-1.4.8-1-centos7.x86_64.rpm --force
Preparing... ################################# [100%]
Updating / installing...1:proxysql-1.4.8-1 ################################# [100%]2、proxysql配置文件详解
[root@proxysql-node ~]# egrep -v "^#|^$" /etc/proxysql.cnf
datadir="/var/lib/proxysql" #数据目录
admin_variables=
{admin_credentials="admin:admin" #连接管理端的用户名与密码mysql_ifaces="0.0.0.0:6032" #管理端口，用来连接proxysql的管理数据库!
}
mysql_variables=
{threads=4 #指定转发端口开启的线程数量max_connections=2048default_query_delay=0default_query_timeout=36000000have_compress=truepoll_timeout=2000interfaces="0.0.0.0:6033" #指定转发端口，用于连接后端真实mysql数据库的，相当于代理作用!default_schema="information_schema"stacksize=1048576server_version="5.5.30" #指定后端mysql的版本connect_timeout_server=3000monitor_username="monitor"monitor_password="monitor"monitor_history=600000monitor_connect_interval=60000monitor_ping_interval=10000monitor_read_only_interval=1500monitor_read_only_timeout=500ping_interval_server_msec=120000ping_timeout_server=500commands_stats=truesessions_sort=trueconnect_retries_on_failure=10
}
mysql_servers =
(
)
mysql_users:
(
)
mysql_query_rules:
(
)
scheduler=
(
)
mysql_replication_hostgroups=
(
)3、启动服务
[root@proxysql-node ~]# /etc/init.d/proxysql start
Starting ProxySQL: DONE!
[root@proxysql-node ~]# ss -anput | grep proxy
tcp LISTEN 0 128 *:6032 *:* users:(("proxysql",pid=22573,fd=23))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=22573,fd=22))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=22573,fd=21))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=22573,fd=20))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=22573,fd=19))4、初始化Proxysql，将之前的proxysql数据都删除
[root@proxysql-node ~]# mysql -uadmin -padmin -h127.0.0.1 -P6032
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.30 (ProxySQL Admin Module)Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]>MySQL [(none)]> show tables;
+--------------------------------------------+
| tables |
+--------------------------------------------+
| global_variables |
| mysql_collations |
| mysql_group_replication_hostgroups |
| mysql_query_rules |
| mysql_query_rules_fast_routing |
| mysql_replication_hostgroups |
| mysql_servers |
| mysql_users |
| proxysql_servers |
| runtime_checksums_values |
| runtime_global_variables |
| runtime_mysql_group_replication_hostgroups |
| runtime_mysql_query_rules |
| runtime_mysql_query_rules_fast_routing |
| runtime_mysql_replication_hostgroups |
| runtime_mysql_servers |
| runtime_mysql_users |
| runtime_proxysql_servers |
| runtime_scheduler |
| scheduler |
+--------------------------------------------+
20 rows in set (0.00 sec)MySQL [(none)]> delete from scheduler ;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> delete from mysql_servers;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> delete from mysql_users;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> delete from mysql_query_rules;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> delete from mysql_group_replication_hostgroups ;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> LOAD MYSQL VARIABLES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL VARIABLES TO DISK;
Query OK, 94 rows affected (0.00 sec)MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.10 sec)MySQL [(none)]> LOAD MYSQL USERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL USERS TO DISK;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.01 sec)MySQL [(none)]> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL QUERY RULES TO DISK;
Query OK, 0 rows affected (0.00 sec)5、在数据库端建立proxysql登入需要的账号（在MGR任意一台，另外的会同步）
[root@mgr-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> CREATE USER 'proxysql'@'%' IDENTIFIED BY 'proxysql';
Query OK, 0 rows affected (0.16 sec)mysql> GRANT ALL ON * . * TO 'proxysql'@'%';
Query OK, 0 rows affected (0.00 sec)mysql> create user 'qyuser'@'%' IDENTIFIED BY 'qypass';
Query OK, 0 rows affected (0.01 sec)mysql> GRANT ALL ON * . * TO 'qyuser'@'%';
Query OK, 0 rows affected (0.10 sec)mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)6、创建检查MGR节点状态的函数和视图（在三个MGR任意一个节点上操作，会自动同步到其他节点）
[root@mgr-node1 ~]# vim /root/addition_to_sys.sql
[root@mgr-node1 ~]# cat /root/addition_to_sys.sql
USE sys;DELIMITER $$CREATE FUNCTION IFZERO(a INT, b INT)
RETURNS INT
DETERMINISTIC
RETURN IF(a = 0, b, a)$$CREATE FUNCTION LOCATE2(needle TEXT(10000), haystack TEXT(10000), offset INT)
RETURNS INT
DETERMINISTIC
RETURN IFZERO(LOCATE(needle, haystack, offset), LENGTH(haystack) + 1)$$CREATE FUNCTION GTID_NORMALIZE(g TEXT(10000))
RETURNS TEXT(10000)
DETERMINISTIC
RETURN GTID_SUBTRACT(g, '')$$CREATE FUNCTION GTID_COUNT(gtid_set TEXT(10000))
RETURNS INT
DETERMINISTIC
BEGINDECLARE result BIGINT DEFAULT 0;DECLARE colon_pos INT;DECLARE next_dash_pos INT;DECLARE next_colon_pos INT;DECLARE next_comma_pos INT;SET gtid_set = GTID_NORMALIZE(gtid_set);SET colon_pos = LOCATE2(':', gtid_set, 1);WHILE colon_pos != LENGTH(gtid_set) + 1 DOSET next_dash_pos = LOCATE2('-', gtid_set, colon_pos + 1);SET next_colon_pos = LOCATE2(':', gtid_set, colon_pos + 1);SET next_comma_pos = LOCATE2(',', gtid_set, colon_pos + 1);IF next_dash_pos < next_colon_pos AND next_dash_pos < next_comma_pos THENSET result = result +SUBSTR(gtid_set, next_dash_pos + 1,LEAST(next_colon_pos, next_comma_pos) - (next_dash_pos + 1)) -SUBSTR(gtid_set, colon_pos + 1, next_dash_pos - (colon_pos + 1)) + 1;ELSESET result = result + 1;END IF;SET colon_pos = next_colon_pos;END WHILE;RETURN result;
END$$CREATE FUNCTION gr_applier_queue_length()
RETURNS INT
DETERMINISTIC
BEGINRETURN (SELECT sys.gtid_count( GTID_SUBTRACT( (SELECT
Received_transaction_set FROM performance_schema.replication_connection_status
WHERE Channel_name = 'group_replication_applier' ), (SELECT
@@global.GTID_EXECUTED) )));
END$$CREATE FUNCTION gr_member_in_primary_partition()
RETURNS VARCHAR(3)
DETERMINISTIC
BEGINRETURN (SELECT IF( MEMBER_STATE='ONLINE' AND ((SELECT COUNT(*) FROM
performance_schema.replication_group_members WHERE MEMBER_STATE != 'ONLINE') >=
((SELECT COUNT(*) FROM performance_schema.replication_group_members)/2) = 0),
'YES', 'NO' ) FROM performance_schema.replication_group_members JOIN
performance_schema.replication_group_member_stats USING(member_id));
END$$CREATE VIEW gr_member_routing_candidate_status AS SELECT
sys.gr_member_in_primary_partition() as viable_candidate,
IF( (SELECT (SELECT GROUP_CONCAT(variable_value) FROM
performance_schema.global_variables WHERE variable_name IN ('read_only',
'super_read_only')) != 'OFF,OFF'), 'YES', 'NO') as read_only,
sys.gr_applier_queue_length() as transactions_behind, Count_Transactions_in_queue as 'transactions_to_cert' from performance_schema.replication_group_member_stats;$$DELIMITER ;
7、导入addition_to_sys.sql文件数据
[root@mgr-node1 ~]# mysql -p123.com < /root/addition_to_sys.sql
mysql: [Warning] Using a password on the command line interface can be insecure.查看该视图：
[root@mgr-node1 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 5.7.31-log MySQL Community Server (GPL)Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> select * from sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
| viable_candidate | read_only | transactions_behind | transactions_to_cert |
+------------------+-----------+---------------------+----------------------+
| YES | NO | 0 | 0 |
+------------------+-----------+---------------------+----------------------+
1 row in set (0.00 sec)8、 在proxysql中增加帐号
[root@proxysql-node ~]# mysql -uadmin -padmin -h127.0.0.1 -P6032
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.30 (ProxySQL Admin Module)Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> INSERT INTO MySQL_users(username,password,default_hostgroup) VALUES ('proxysql','proxysql',1);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> UPDATE global_variables SET variable_value='proxysql' where variable_name='mysql-monitor_username';
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> UPDATE global_variables SET variable_value='proxysql' where variable_name='mysql-monitor_password';
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.00 sec)
9、登陆测试
如果上面测试登录时报错:
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h 127.0.0.1 -P6033 -e"select @@hostname"
ERROR 1045 (28000): ProxySQL Error: Access denied for user 'proxysql'@'127.0.0.1' (using password: YES)但是检查发现，明明用户名和密码已经修改成proxysql:proxysql了
MySQL [(none)]> select * from global_variables;
+-----------------------------------------------------+--------------------+
| variable_name | variable_value |
+-----------------------------------------------------+--------------------+
.
.
.
| mysql-monitor_username | proxysql |
| mysql-monitor_password | proxysql |
| mysql-monitor_history | 600000 |
| mysql-monitor_connect_interval | 60000 |
.
.
| mysql-commands_stats | true |
| mysql-sessions_sort | true |
| mysql-connect_retries_on_failure | 10 |
+-----------------------------------------------------+--------------------+执行下面操作
将上面对proxysql所有的变更都加载到环境中
MySQL [(none)]> LOAD MYSQL VARIABLES TO RUNTIME;
Query OK, 0 rows affected (0.01 sec)MySQL [(none)]> SAVE MYSQL VARIABLES TO DISK;
Query OK, 94 rows affected (0.01 sec)MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.01 sec)MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.10 sec)MySQL [(none)]> LOAD MYSQL USERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL USERS TO DISK;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL QUERY RULES TO DISK;
Query OK, 0 rows affected (0.01 sec)MySQL [(none)]> delete from mysql_servers;
Query OK, 0 rows affected (0.00 sec)将后端三个mysql的MGR节点加入到proxysql中MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.153.149',3306);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.153.150',3306);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.153.151',3306);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.153.149',3306);
Query OK, 1 row affected (0.01 sec)MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.153.150',3306);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.153.151',3306);
Query OK, 1 row affected (0.01 sec)查看结果
MySQL [(none)]> select * from mysql_servers ;
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
hostgroup_id = 1代表write group，针对我们提出的限制，这个地方只配置了一个节点；
hostgroup_id = 2代表read group，包含了MGR的所有节点,目前只是Onlinle的，等配置过scheduler后，status就会有变化 。
对于上面的hostgroup配置，默认所有的写操作会发送到hostgroup_id为1的online节点，也就是发送到写节点上。
所有的读操作，会发送为hostgroup_id为2的online节点。
10、验证proxysql登录
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h 127.0.0.1 -P6033 -e"select @@hostname"
+------------+
| @@hostname |
+------------+
| mgr-node3 |
+------------+
11、配置scheduler
上传脚本
[root@proxysql-node ~]# ls
anaconda-ks.cfg proxysql-1.4.8-1-centos7.x86_64.rpm proxysql_groupreplication_checker-master.zip将脚本proxysql_groupreplication_checker.sh解压放到/var/lib/proxysql/下
[root@proxysql-node ~]# cd /var/lib/proxysql/
[root@proxysql-node proxysql]# ls
proxysql.db proxysql.log proxysql.pid proxysql_stats.db
[root@proxysql-node proxysql]# mv /root/proxysql_groupreplication_checker-master.zip .
[root@proxysql-node proxysql]# ls
proxysql.db proxysql_groupreplication_checker-master.zip proxysql.log proxysql.pid proxysql_stats.db
[root@proxysql-node proxysql]# unzip proxysql_groupreplication_checker-master.zip
Archive: proxysql_groupreplication_checker-master.zip
1f34bf1a67da5a26a0c6e16dfb74349871e6470fcreating: proxysql_groupreplication_checker-master/inflating: proxysql_groupreplication_checker-master/README.mdinflating: proxysql_groupreplication_checker-master/README_Chinese.mdinflating: proxysql_groupreplication_checker-master/gr_mw_mode_sw_cheker.shinflating: proxysql_groupreplication_checker-master/gr_sw_mode_checker.shinflating: proxysql_groupreplication_checker-master/proxysql_groupreplication_checker.sh
[root@proxysql-node proxysql]# ls
proxysql.db proxysql_groupreplication_checker-master.zip proxysql.pid
proxysql_groupreplication_checker-master proxysql.log proxysql_stats.db
[root@proxysql-node proxysql_groupreplication_checker-master]# mv * /var/lib/proxysql/
[root@proxysql-node proxysql]# chmod a+x /var/lib/proxysql/proxysql_groupreplication_checker.sh
[root@proxysql-node proxysql]# ll /var/lib/proxysql/proxysql_groupreplication_checker.sh
-rwxr-xr-x 1 root root 6081 Jan 9 2017 /var/lib/proxysql/proxysql_groupreplication_checker.sh然后，在proxysql的scheduler表里面加载如下记录，然后加载到RUNTIME使其生效，同时还可以持久化到磁盘:[root@proxysql-node proxysql]# mysql -uadmin -padmin -h127.0.0.1 -P6032
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.5.30 (ProxySQL Admin Module)Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> INSERT INTO scheduler(id,interval_ms,filename,arg1,arg2,arg3,arg4, arg5) VALUES (1,'10000','/var/lib/proxysql/ proxysql_groupreplication_checker.sh','1','2','1','0','/var/lib/proxysql/proxysql_groupreplication_checker.log');
Query OK, 1 row affected (0.00 sec)查看结果
MySQL [(none)]> select * from scheduler;
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| id | active | interval_ms | filename | arg1 | arg2 | arg3 | arg4 | arg5 | comment |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| 1 | 1 | 10000 | /var/lib/proxysql/proxysql_groupreplication_checker.sh | 1 | 2 | 1 | 0 | /var/lib/proxysql/proxysql_groupreplication_checker.log | |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
1 row in set (0.00 sec)MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.01 sec)schedule信息加载后，就会分析当前的环境，mysql_servers中显示出当前只有192.168.153.149是可以写的192.168.153.150以及192.168.153.151是用来读的。
MySQL [(none)]> select * from mysql_servers ;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.150 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.151 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.149 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)各个节点的gr_member_routing_candidate_status视图也显示了当前节点是否是正常状态的，
proxysql就是读取的这个视图的信息来决定此节点是否可用。
mysql> select * from sys.gr_member_routing_candidate_status\G;
*************************** 1. row ***************************viable_candidate: YESread_only: NOtransactions_behind: 0
transactions_to_cert: 0
1 row in set (0.00 sec)ERROR:
No query specifiedmysql>
12、设置读写分离
MySQL [(none)]> insert into mysql_query_rules (active, match_pattern, destination_hostgroup, apply) values (1,"^SELECT",2,1);
Query OK, 1 row affected (0.00 sec)MySQL [(none)]> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)MySQL [(none)]> SAVE MYSQL QUERY RULES TO DISK;
Query OK, 0 rows affected (0.01 sec)解释说明:
match_pattern的规则是基于正则表达式的，
active表示是否启用这个sql路由项，
match_pattern就是我们正则匹配项，
destination_hostgroup表示我们要将该类sql转发到哪些mysql上面去，这里我们将select转发到group 2，。
apply为1表示该正则匹配后，将不再接受其他匹配，直接转发。对于for update需要在gruop1上执行，可以加上规则：
MySQL [(none)]> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values(1,'^SELECT.*FOR UPDATE$',1,1);
Query OK, 1 row affected (0.01 sec)在proxysql本机或其他客户机上检查下,select 语句,一直连接的是192.168.153.150和192.168.153.151
说明读写分离[root@mgr-node3 ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| mgr-node2 |
+------------+
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| mgr-node2 |
+------------+
[root@mgr-node3 ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| mgr-node3 |
+------------+
[root@mgr-node3 ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| mgr-node3 |
+------------+
[root@mgr-node3 ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| mgr-node2 |
+------------+验证读写分离的效果[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| mgr-node2 |
+------------+
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select * from kevin.haha"
+----+-----------+
| id | name |
+----+-----------+
| 1 | wangshibo |
| 2 | guohuihui |
| 3 | yangyang |
| 4 | shikui |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+-----------+
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "delete from kevin.haha where id=1;"
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "delete from kevin.haha where id=2;"
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select * from kevin.haha"
+----+----------+
| id | name |
+----+----------+
| 3 | yangyang |
| 4 | shikui |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+----------+[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e 'insert into kevin.haha values(21,"zhongguo"),(22,"xianggang"),(23,"taiwan");'
[root@proxysql-node ~]# mysql -uproxysql -pproxysql -h192.168.153.152 -P6033 -e "select * from kevin.haha"
+----+-----------+
| id | name |
+----+-----------+
| 3 | yangyang |
| 4 | shikui |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
| 21 | zhongguo |
| 22 | xianggang |
| 23 | taiwan |
+----+-----------+最后在proxysql管理端查看读写分离情况[root@proxysql-node ~]# mysql -uadmin -padmin -h 127.0.0.1 -P6032
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1575
Server version: 5.5.30 (ProxySQL Admin Module)Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MySQL [(none)]> select hostgroup,username,digest_text,count_star from stats_mysql_query_digest;
+-----------+----------+------------------------------------------------+------------+
| hostgroup | username | digest_text | count_star |
+-----------+----------+------------------------------------------------+------------+
| 1 | proxysql | delete from kevin.haha where id=? | 2 |
| 1 | proxysql | insert into kevin.haha values(?,?),(?,?),(?,?) | 1 |
| 2 | proxysql | select * from kevin.haha | 3 |
| 2 | proxysql | select @@hostname | 5 |
| 1 | proxysql | select @@hostname | 1 |
| 1 | proxysql | select @@version_comment limit ? | 12 |
+-----------+----------+------------------------------------------------+------------+
6 rows in set (0.00 sec)MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.150 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.151 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.149 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
通过上面可以看到：
写操作都分配到了group1组内，即写操作分配到192.168.153.149节点上。
读操作都分配到了group2组内，即读操作分配到192.168.153.150、192.168.153.151节点上。13、设置故障应用无感应
在上面的读写分离规则中，我设置了192.168.153.149为可写节点，192.168.153.150,192.168.153.151为只读节点。如果此时192.168.153.149变成只读模式的话，应用能不能直接连到其它的节点进行写操作？
1、手动将192.168.153.149变成只读模式：
mysql> set global read_only=1;
Query OK, 0 rows affected (0.00 sec)
2、在proxysql节点上查看，mysql_servers的状态，自动将group1的192.168.153.150改成了online，group2的192.168.153.149,192.168.153.151变成online了，就表示将192.168.153.150变为可写节点，其它两个节点变为只读节点了。
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.153.149 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.151 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.150 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)然后再将将192.168.153.149变为可写模式后，mysql_servers也恢复过来了。mysql> set global read_only=0;
Query OK, 0 rows affected (0.00 sec)然后在proxysql节点查看
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.153.149 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.150 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.153.151 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.149 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.150 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.153.151 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)经过测试将192.168.153.149节点停止组复制（stop group_replication）或者该节点宕机(mysql服务挂掉)后，mysql_servers表的信息也会正常的切换新的节点。
待192.168.153.149恢复再加入到组复制后，mysql_servers也会正常的将192.168.153.149改成online状态。ProxySQL+MGR实现读写分离和主节点故障无感知切换相关推荐
- ProxySQL+MGR实现读写分离和主节点故障无感知切换 - 完整操作记录
ProxySQL+MGR实现读写分离和主节点故障无感知切换 - 完整操作记录 前面的文章介绍了ProxySQL用法,这里说下ProxySQL中间件针对Mysql组复制模式实现读写分离以及主节点故障时能 ...
- MySQL用中间件ProxySQL实现读写分离和主节点故障应用无感应
昨天做的用proxysql实现的读写分离,但是在实际的应用中这样的结构还很不完整,如果主节点出现故障那么整个拓扑的数据库也无法通过proxysql来调用了,所以还需要增加主节点故障后proxysql能 ...
- MySQL高可用实现：主从结构下ProxySQL中的读写分离
墨墨导读:ProxySQL是一个高性能的MySQL中间件,拥有强大的规则引擎. ProxySQL提供强大的路由规则.当应用程序自身不支持读写分离时,DBA可以通过配置路由规则为应用程序提供透明的读写分 ...
- Mycat读写分离案例及故障转移
1.概述 1.1读写分离 数据某个分片中,实现的主从高可用备份,通过对主的写操作,对从的读操作将读写分离执行提升集群使用效率 1.2故障转移 一个数据分片的读写工作,由于正在使用的某个节点出现故障,连 ...
- Linux随笔19-MySQL主从复制、Percona XtraBackup实现全量和增量备份、ProxySQL实现读写分离
Contents 1. MySQL5.7实现主从复制 1.1 基础环境 1.2. 配置主从复制 1.2.1. master节点上的配置 1.2.2. slave节点上的配置 1.2.3. 中继日志问题 ...
- 【DB宝42】MySQL高可用架构MHA+ProxySQL实现读写分离和负载均衡
文章目录 一.MHA+ProxySQL架构 二.快速搭建MHA环境 2.1 下载MHA镜像 2.2 编辑yml文件,创建MHA相关容器 2.3 安装docker-compose软件(若已安装,可忽略) ...
- ProxySQL 配置详解及读写分离(+GTID)等功能说明 (完整篇)2
2) 从数据库mysql-slave1 (172.16.60.212)的配置操作 与主服务器配置大概一致,除了server_id不一致外,从服务器还可以在配置文件里面添加:"read_onl ...
- ProxySQL 配置详解及读写分离(+GTID)等功能说明2 (完整篇)
1. 实验环境 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 ...
- ProxySQL 配置详解及读写分离(+GTID)等功能说明 (完整篇)
ProxySQL是灵活强大的MySQL代理层, 是一个能实实在在用在生产环境的MySQL中间件,可以实现读写分离,支持 Query 路由功能,支持动态指定某个 SQL 进行 cache,支持动态加载配 ...
最新文章
- boost::fusion::make_map用法的测试程序
- react中实现异步请求的方法一，react-thunk
- Memory及其controller芯片整体测试方案（上篇）
- 能用10块大洋去解决的事，千万不要用时间
- 晚上没有路灯，我骑的很慢
- iptables联系一
- 这5款功能强大、堪称最良心的浏览器，简直好用极了！
- 8000401a 因为配置标识不正确,系统无法开始服务器进程。请检查用户名和密码
- houdini 基础
- 4g/5g信号远程遥控小车解决方案二
- ros buntu安装手册_超详细 ROS安装教程
- SQL注入SQLmap简单用法，和SQL注入写入一句话木马
- Tomcat配置登录拦截功能
- Python的PyOpenGL
- The application's PagerAdapter changed the adapter's contents without calling PagerAdapter
- 很多人已经学会了应对恐惧和焦虑的秘诀
- 出门问问变阵 | 中国独角兽豹变之路
- 从0到100： 基于微信云开发的驾校预约学车小程序
- 关于移动支付，美国零售商能从中国这里学到什么？
- EBS R12多组织的实现原理