Doris集群环境搭建

本文记录按照官网文档进行操作过程

1、下载预编译包

（1）百度Doris官方网址
2021年，百度网址doris.baidu.com/doc已经调整为了
https://cloud.baidu.com/doc/PALO/s/Wksis5irl

请注意官方提示：

适用于CentOS 7.3
依赖Oracle JDK 1.8 版本

（2）确定CPU是否支持avx2指令

检查如下，我使用的是虚拟机，暂不支持avx2指令，因此需要下载对应的no-avx2版本的预编译

[root@node3 ~]# cat /proc/cpuinfo | grep flags
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
[root@node3 ~]# cat /proc/cpuinfo | grep avx2
[root@node3 ~]#

（3）下载
如上图，单击0.14.13-no-avx2后面的Link，进行下载
https://palo-cloud-repo-bd.bd.bcebos.com/baidu-doris-release/PALO-0.14.13-release-no-avx2-binary.tar.gz

2、FE基本配置

（1）将软件包上传到服务器
此处我上传到了自己创建的/app目录下

（2）加压并重命名

[root@node3 app]# tar -zxvf PALO-0.14.13-release-no-avx2-binary.tar.gz
[root@node3 app]# mv PALO-0.14.13-release-no-avx2-binary doris-0.14.13
[root@node3 app]# cd doris-0.14.13/
[root@node3 doris-0.14.13]# ls
apache_hdfs_broker  auditloader.zip  be  doris-flink-1.0-SNAPSHOT.jar  doris-spark-1.0.0-SNAPSHOT.jar  fe  udf
[root@node3 doris-0.14.13]#

（3）创建FE节点元数据目录

[root@node3 doris-0.14.13]# cd fe
[root@node3 fe]# mkdir doris-meta
[root@node3 fe]#

（4）配置参数

[root@node3 fe]# vi conf/fe.conf

修改JAVA_OPTS参数，默认内存是4G，根据具体情况修改。生产环境建议8G以上，我这里是测试环境，修改为了1G（1024M）

JAVA_OPTS="-Xmx1024m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

配置priority_networks

priority_networks = 10.17.12.160/24

3、BE基本配置

（1）配置环境参数/etc/security/limits.conf
如果这一步已经配置过了，可以跳过。

[root@node1 be]# vi /etc/security/limits.conf

在文件末尾添加如下内容

# End of file* hard nofile 65536
* soft nofile 65536

（2）创建数据目录

[root@node3 fe]# cd ../be
[root@node3 be]# mkdir storage
[root@node3 be]#

（3）修改配置文件

[root@node3 be]# vi conf/be.conf

priority_networks = 10.17.12.160/24

4、配置集群节点

（1）复制软件包到集群节点

[root@node3 app]# scp -r doris-0.14.13/ node1:/app
[root@node3 app]# scp -r doris-0.14.13/ node2:/app

（2）修改FE配置
先修改node1

[root@node1 ~]# cd /app/doris-0.14.13/
[root@node1 doris-0.14.13]# vi fe/conf/fe.conf
[root@node1 doris-0.14.13]#

priority_networks = 10.17.12.158/24

在修改node2

[root@node2 ~]# cd /app/doris-0.14.13/
[root@node2 doris-0.14.13]# vi fe/conf/fe.conf
[root@node2 doris-0.14.13]#

priority_networks = 10.17.12.159/24

（3）修改BE配置

[root@node1 doris-0.14.13]# vi be/conf/be.conf

[root@node2 doris-0.14.13]# vi be/conf/be.conf

修改内容：
node1:

priority_networks = 10.17.12.158/24

node2

priority_networks = 10.17.12.159/24

注意：如果是第一次配置了/etc/security/limits.conf，请重启机器。

5、启动FE 集群

（1）启动第一个FE节点

[root@node1 app]# cd doris-0.14.13/fe
[root@node1 fe]# sh bin/start_fe.sh --daemon
[root@node1 fe]# jps
10400 PaloFe
12744 Worker
12249 Master
10475 Jps
[root@node1 fe]#

（2）登录客户端
参考https://blog.csdn.net/chengyuqiang/article/details/119974926按照mysql-client客户端

[root@node1 fe]# mysql -h10.17.12.158 -P9030 -uroot
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 0
Server version: 5.1.0Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql>

（3）添加FE节点

mysql> ALTER SYSTEM ADD FOLLOWER "node2:9010";
Query OK, 0 rows affected (0.04 sec)mysql> ALTER SYSTEM ADD FOLLOWER "node3:9010";
Query OK, 0 rows affected (0.01 sec)mysql>

（4）查看FE集群状态

mysql> show frontends \G
*************************** 1. row ***************************Name: 10.17.12.158_9010_1630292462734IP: 10.17.12.158EditLogPort: 9010HttpPort: 8030QueryPort: 9030RpcPort: 9020Role: FOLLOWERIsMaster: trueClusterId: 1583270145Join: trueAlive: true
ReplayedJournalId: 219LastHeartbeat: 2021-08-30 11:13:04IsHelper: trueErrMsg: Version: 0.14.13-release-Unknown
*************************** 2. row ***************************Name: 10.17.12.160_9010_1630293081245IP: 10.17.12.160EditLogPort: 9010HttpPort: 8030QueryPort: 0RpcPort: 0Role: FOLLOWERIsMaster: falseClusterId: 1583270145Join: falseAlive: false
ReplayedJournalId: 0LastHeartbeat: NULLIsHelper: trueErrMsg: got exceptionVersion: NULL
*************************** 3. row ***************************Name: 10.17.12.159_9010_1630293077685IP: 10.17.12.159EditLogPort: 9010HttpPort: 8030QueryPort: 0RpcPort: 0Role: FOLLOWERIsMaster: falseClusterId: 1583270145Join: falseAlive: false
ReplayedJournalId: 0LastHeartbeat: NULLIsHelper: trueErrMsg: got exceptionVersion: NULL
3 rows in set (0.06 sec)mysql>

（5）启动其他两个FE节点

[root@node2 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
[root@node2 fe]# jps
6144 Worker
18516 PaloFe
18568 Jps
[root@node2 fe]#

[root@node3 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
[root@node3 fe]# jps
5971 PaloFe
1940 Worker
6025 Jps
[root@node3 fe]#

mysql> show frontends \G
*************************** 1. row ***************************Name: 10.17.12.158_9010_1630292462734IP: 10.17.12.158EditLogPort: 9010HttpPort: 8030QueryPort: 9030RpcPort: 9020Role: FOLLOWERIsMaster: trueClusterId: 1583270145Join: trueAlive: true
ReplayedJournalId: 286LastHeartbeat: 2021-08-30 11:16:50IsHelper: trueErrMsg: Version: 0.14.13-release-Unknown
*************************** 2. row ***************************Name: 10.17.12.160_9010_1630293081245IP: 10.17.12.160EditLogPort: 9010HttpPort: 8030QueryPort: 9030RpcPort: 9020Role: FOLLOWERIsMaster: falseClusterId: 1583270145Join: trueAlive: true
ReplayedJournalId: 285LastHeartbeat: 2021-08-30 11:16:50IsHelper: trueErrMsg: Version: 0.14.13-release-Unknown
*************************** 3. row ***************************Name: 10.17.12.159_9010_1630293077685IP: 10.17.12.159EditLogPort: 9010HttpPort: 8030QueryPort: 9030RpcPort: 9020Role: FOLLOWERIsMaster: falseClusterId: 1583270145Join: trueAlive: true
ReplayedJournalId: 285LastHeartbeat: 2021-08-30 11:16:50IsHelper: trueErrMsg: Version: 0.14.13-release-Unknown
3 rows in set (0.03 sec)mysql>

6、启动BE集群

（1）添加BE节点

mysql> ALTER SYSTEM ADD BACKEND "node1:9050";
Query OK, 0 rows affected (0.04 sec)mysql> ALTER SYSTEM ADD BACKEND "node2:9050";
Query OK, 0 rows affected (0.02 sec)mysql> ALTER SYSTEM ADD BACKEND "node3:9050";
Query OK, 0 rows affected (0.01 sec)mysql>

（2）查看BE集群状态

mysql> show backends \G
*************************** 1. row ***************************BackendId: 10002Cluster: default_clusterIP: 10.17.12.158HeartbeatPort: 9050BePort: -1HttpPort: -1BrpcPort: -1LastStartTime: NULLLastHeartbeat: NULLAlive: falseSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 1.000 BTotalCapacity: .000 UsedPct: 0.00 %MaxDiskUsedPct: 0.00 %ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)Version: Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
*************************** 2. row ***************************BackendId: 10003Cluster: default_clusterIP: 10.17.12.159HeartbeatPort: 9050BePort: -1HttpPort: -1BrpcPort: -1LastStartTime: NULLLastHeartbeat: NULLAlive: falseSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 1.000 BTotalCapacity: .000 UsedPct: 0.00 %MaxDiskUsedPct: 0.00 %ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)Version: Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
*************************** 3. row ***************************BackendId: 10004Cluster: default_clusterIP: 10.17.12.160HeartbeatPort: 9050BePort: -1HttpPort: -1BrpcPort: -1LastStartTime: NULLLastHeartbeat: NULLAlive: falseSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 1.000 BTotalCapacity: .000 UsedPct: 0.00 %MaxDiskUsedPct: 0.00 %ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)Version: Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
3 rows in set (0.01 sec)mysql>

这是因为当前3个BE节点都没有启动

（3）启动BE节点

[root@node1 ~]# cd /app/doris-0.14.13/be
[root@node1 be]# sh bin/start_be.sh --daemon
[root@node1 be]#  ps -aux |grep _be
root     11402  2.3  1.3 2875316 111728 pts/2  Sl   11:23   0:00 /app/doris-0.14.13/be/lib/palo_be
root     12344  0.0  0.0 112664   956 pts/2    S+   11:23   0:00 grep --color=auto _be
[root@node1 be]#

[root@node2 be]# pwd
/app/doris-0.14.13/be
[root@node2 be]# sh bin/start_be.sh --daemon
[root@node2 be]# ps -aux |grep _be
root     27454  0.6  1.4 2876336 114208 pts/0  Sl   11:21   0:00 /app/doris-0.14.13/be/lib/palo_be
root     29140  0.0  0.0 112664   960 pts/0    S+   11:22   0:00 grep --color=auto _be
[root@node2 be]#

[root@node3 be]# pwd
/app/doris-0.14.13/be
[root@node3 be]# sh bin/start_be.sh --daemon
[root@node3 be]# ps -aux |grep _be
root     15805  1.6  1.1 2876336 95692 pts/0   Sl   11:22   0:00 /app/doris-0.14.13/be/lib/palo_be
root     16473  0.0  0.0 112668   960 pts/0    S+   11:22   0:00 grep --color=auto _be
[root@node3 be]#

（4）稍等片刻，再次查看BE集群状态

mysql> show backends \G
*************************** 1. row ***************************BackendId: 10002Cluster: default_clusterIP: 10.17.12.158HeartbeatPort: 9050BePort: 9060HttpPort: 8040BrpcPort: 8060LastStartTime: 2021-08-30 11:23:50LastHeartbeat: 2021-08-30 11:25:56Alive: trueSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 6.456 GBTotalCapacity: 49.090 GBUsedPct: 86.85 %MaxDiskUsedPct: 86.85 %ErrMsg: Version: 0.14.13-release-UnknownStatus: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:52","lastStreamLoadTime":-1}
*************************** 2. row ***************************BackendId: 10003Cluster: default_clusterIP: 10.17.12.159HeartbeatPort: 9050BePort: 9060HttpPort: 8040BrpcPort: 8060LastStartTime: 2021-08-30 11:21:53LastHeartbeat: 2021-08-30 11:25:56Alive: trueSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 9.921 GBTotalCapacity: 49.090 GBUsedPct: 79.79 %MaxDiskUsedPct: 79.79 %ErrMsg: Version: 0.14.13-release-UnknownStatus: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:56","lastStreamLoadTime":-1}
*************************** 3. row ***************************BackendId: 10004Cluster: default_clusterIP: 10.17.12.160HeartbeatPort: 9050BePort: 9060HttpPort: 8040BrpcPort: 8060LastStartTime: 2021-08-30 11:22:04LastHeartbeat: 2021-08-30 11:25:56Alive: trueSystemDecommissioned: false
ClusterDecommissioned: falseTabletNum: 0DataUsedCapacity: .000 AvailCapacity: 7.602 GBTotalCapacity: 49.090 GBUsedPct: 84.51 %MaxDiskUsedPct: 84.51 %ErrMsg: Version: 0.14.13-release-UnknownStatus: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:07","lastStreamLoadTime":-1}
3 rows in set (0.01 sec)mysql>