Kafka集群搭建实战

集群结构：

集群环境信息：

192.168.157.130
192.168.157.131
192.168.157.129

Zookeeper集群搭建

三台机器上均安装JDK（这里采用压缩包的安装方式jdk-8u301-linux-x64.tar.gz）

xxx:~#tar -zxvf jdk-8u301-linux-x64.tar.gz # 解压文件
xxx:~#mv jdk1.8.0_301 /opt                 # 移动文件至 /opt目录下
xxx:~#vim /etc/profile                     # 编辑 /etc/profile 文件
# jdk
export JAVA_HOME=/opt/jdk1.8.0_301
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=.:${JAVA_HOME}/bin:$PATH
xxx:~#source /etc/profile                   # 用source命令使新加的配置生效
xxx:~#java -version                         # java -version 验证java是否配置成功
java version "1.8.0_301"
Java(TM) SE Runtime Environment (build 1.8.0_301-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)

三台机器上均安装Zookeeper用来搭建集群（这里采用压缩包的安装方式zookeeper-3.4.14.tar.gz）

第一台机器上配置

xxx:~#tar -zxf zookeeper-3.4.14.tar.gz      # 解压文件
xxx:~#mv zookeeper-3.4.14 /opt              # 移动文件至 /opt 目录下
xxx:~#cd /opt/zookeeper-3.4.14/conf         # 切换目录至 /opt/zookeeper-3.4.14/conf
xxx:/opt/zookeeper-3.4.14/conf#cp zoo_sample.cfg zoo.cfg # 复制 zoo_sample.cfg 配置文件为 zoo.cfg
xxx:/opt/zookeeper-3.4.14/conf#vim zoo.cfg  # 编辑 zoo.cfg
# 设置
dataDir=/var/software/zookeeper/data
# 添加
server.1=192.168.157.130:2881:3881
server.2=192.168.157.131:2881:3881
server.3=192.168.157.129:2881:3881
xxx:/opt/zookeeper-3.4.14/conf#mkdir -p /var/software/zookeeper/data # 创建目录/var/software/zookeeper/data
xxx:/opt/zookeeper-3.4.14/conf#echo 1 > /var/software/zookeeper/data/myid # 写入 1 至 文件
xxx:~#vim /etc/profile                      # 编辑 /etc/profile 文件
# zookeeper
export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/software/zookeeper/log
xxx:~#source /etc/profile                   # 用 source 命令使配置生效
xxx:~#scp -r /opt/zookeeper-3.4.14/ 192.168.157.131:/opt # 用 scp 命令远程传输文件至指定主机下的指定目录
xxx:~#scp -r /opt/zookeeper-3.4.14/ 192.168.157.129:/opt # 同上

第二台机器上配置

xxx:~#vim /etc/profile
# zookeeper
export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/software/zookeeper/log
xxx:~#source /etc/profile
xxx:~#mkdir -p /var/software/zookeeper/data
xxx:~#echo 2 > /var/software/zookeeper/data/myid

第三台机器上配置

xxx:~#vim /etc/profile
# zookeeper
export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/software/zookeeper/log
xxx:~#source /etc/profile
xxx:~#mkdir -p /var/software/zookeeper/data
xxx:~#echo 3 > /var/software/zookeeper/data/myid

启动zookeeper（三台机器均按如下操作）
```
xxx:~#zkServer.sh start
xxx:~#zkServer.sh status
```

Kafka集群搭建

三台机器上安装Kafka

第一台机器上配置

xxx:~#tar -zvxf kafka_2.12-1.0.2.tgz
xxx:~#mv kafka_2.12-1.0.2 /opt
xxx:~#scp -r /opt/kafka_2.12-1.0.2/ 192.168.157.131:/opt
xxx:~#scp -r /opt/kafka_2.12-1.0.2/ 192.168.157.129:/opt
xxx:~#vim /etc/profile
# kafka
export KAFKA_HOME=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA_HOME/bin
xxx:~#source /etc/profile
xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.157.130:9092
log.dirs=/var/software/kafka/log
zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka

第二台机器上配置

xxx:~#vim /etc/profile
# kafka
export KAFKA_HOME=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA_HOME/bin
xxx:~#source /etc/profile
xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.157.131:9092
log.dirs=/var/software/kafka/log
zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka

第三台机器上配置

xxx:~#vim /etc/profile
# kafka
export KAFKA_HOME=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA_HOME/bin
xxx:~#source /etc/profile
xxx:~#vim /opt/kafka_2.12-1.0.2/config/server.properties
broker.id=2
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://192.168.157.129:9092
log.dirs=/var/software/kafka/log
zookeeper.connect=192.168.157.130:2181,192.168.157.131:2181,192.168.157.129:2181/myKafka

启动Kafka（三台机器均按如下操作）

xxx:~#kafka-server-start.sh /opt/kafka_2.12-1.0.2/config/server.properties

验证Kafka（查看三台机器的Kafka启动过程中的Cluster ID是否一致）

Cluster Id是一个唯一的不可变的标志符，用于唯一标志一个Kafka集群。
该Id最多可以有22个字符组成，字符对应于URL-safe Base64。
Kafka 0.10.1版本及之后的版本中，在集群第一次启动的时候，Broker从Zookeeper

的<Kafka_ROOT>/cluster/id节点获取。如果该Id不存在，就自动生成一个新的。

xxx:~#zkCli.sh
[zk: localhost:2181(CONNECTED) 2] ls /myKafka/brokers/ids
[0, 1, 2]
[zk: localhost:2181(CONNECTED) 3] get /myKafka/brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.130:9092"],"jmx_port":-1,"host":"192.168.157.130","timestamp":"1629859078697","port":9092,"version":4}
cZxid = 0x200000097
ctime = Wed Aug 25 10:37:58 CST 2021
mZxid = 0x200000097
mtime = Wed Aug 25 10:37:58 CST 2021
pZxid = 0x200000097
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x200004838ed000a
dataLength = 200
numChildren = 0
[zk: localhost:2181(CONNECTED) 4] get /myKafka/brokers/ids/1
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.131:9092"],"jmx_port":-1,"host":"192.168.157.131","timestamp":"1629859267652","port":9092,"version":4}
cZxid = 0x20000010d
ctime = Wed Aug 25 10:41:07 CST 2021
mZxid = 0x20000010d
mtime = Wed Aug 25 10:41:07 CST 2021
pZxid = 0x20000010d
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x300004845880007
dataLength = 200
numChildren = 0
[zk: localhost:2181(CONNECTED) 5] get /myKafka/brokers/ids/2
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://192.168.157.129:9092"],"jmx_port":-1,"host":"192.168.157.129","timestamp":"1629859117629","port":9092,"version":4}
cZxid = 0x2000000f8
ctime = Wed Aug 25 10:38:37 CST 2021
mZxid = 0x2000000f8
mtime = Wed Aug 25 10:38:37 CST 2021
pZxid = 0x2000000f8
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x10000751a660004
dataLength = 200
numChildren = 0

Kafka集群搭建实战相关推荐

Centos7 安装ELK/EFK 7.x 集群搭建实战攻略
Centos7 安装ELK/EFK 7.x 集群搭建实战攻略 1.1 写在前面的话 1.2 开始之前 1.3 安装ELK/EFK 1.3.1 海量数据存储和检索-->E-->安装Elast ...
转-Kafka【第一篇】Kafka集群搭建
转自: https://www.cnblogs.com/luotianshuai/p/5206662.html Kafka[第一篇]Kafka集群搭建 Kafka初识 1.Kafka使用背景在我们大 ...
kafka集群搭建(消息)
1.Kafka使用背景在我们大量使用分布式数据库.分布式计算集群的时候,是否会遇到这样的一些问题: 我们想分析下用户行为(pageviews),以便我们设计出更好的广告位我想对用户的搜索关键词进行 ...
zookeeper 和 kafka 集群搭建
Kafka初识 1.Kafka使用背景在我们大量使用分布式数据库.分布式计算集群的时候,是否会遇到这样的一些问题: 我们想分析下用户行为(pageviews),以便我们设计出更好的广告位我想对用户 ...
Zookeeper+Kafka集群搭建
Zookeeper集群搭建 Kafka集群是把状态保存在Zookeeper中的,首先要搭建Zookeeper集群. 1.软件环境 (3台服务器-我的测试) 192.168.30.204 server1 ...
kafka集群搭建教程（使用自带的zookeeper）
kafka集群搭建教程(使用自带的zookeeper) 一.kafka简介二.kafka名词解释三.zookeeper与Kafka 四.kafka集群搭建前准备 1.下载 2.检验jdk 3.系统 ...
kafka集群搭建+权限认证（SASL/SCRAM）+整合springboot
本文介绍的的是kafka集群搭建.kafka权限认证(SASL/SCRAM).整合springboot项目. 1.创建kafka日志和zookeeper文件目录: /data/kafka/kafka- ...
Kafka集群搭建及SASL/SCRAM,ACL权限控制
Kafka集群搭建及SASL/SCRAM,ACL权限控制环境部署zookeeper集群及安全认证部署kafka集群及安全认证环境系统版本:Centos7.4 kafka版本:kafka_2. ...
Windows下Kafka集群搭建
一.Windows下Zookeeper集群搭建. 1.集群版本:2.8.1,3.0版本不再支持JDK8,不在需要Zookeeper. 2.Zookeeper版本:3.8.0. 3.Zookeeper三 ...

Kafka集群搭建实战

Kafka集群搭建实战

Zookeeper集群搭建

Kafka集群搭建

Kafka集群搭建实战相关推荐

最新文章

热门文章