ZK实现SASL认证+Kafka连接ZK

关键词:zk sasl kfaka 未授权认证

参考文档:

• 搭建Kafka集群时, 对ZooKeeper认证与权限控制
  http://ohmycat.me/2019/05/08/kafka-with-zookeeper-authentication.html
• SASL authentication for ZooKeeper.
  https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL

漏洞描述

ZooKeeper 未授权访问【原理扫描】
在通常情况下，zookeeper允许未经授权的访问
为ZooKeeper配置相应的访问权限。

服务器IP列表:

• 192.168.132.11
• 192.168.132.12
• 192.168.132.13

镜像版本:

• zookeeper:3.6.1
• wurstmeister/kafka:2.13-2.6.0

1. zookeeper

1.1 设置环境变量

# 备份配置文件
cp -p ~/.bashrc ~/.bashrc.20201218# 编辑~/.bashrc,在文件尾部增加如下内容:
ls_iface=`ls -l /sys/class/net/ |grep -v virtual |grep root|gawk '{print $9}'`
MYIP=`ip -h -4 -o address |grep $ls_iface|gawk '{print $4}'|sed 's/\// /'|gawk '{print $1}'`
IMAGE_NAME=`echo $MYIP |gawk -F. '{print $4}'`
export MYIP IMAGE_NAME# 为每台服务器设置ZKID。
# 注意：每台服务器的ID必需不同且与配置文件中的ZOO_SERVERS的server.x的x要匹配
export myid=1

1.2 让环境变量生效

source ~/.bashrc# 检查环境变量是否效
echo "IP:$MYIP 容器序号:$IMAGE_NAME 服务器ID:$myid"

1.3 (可选)设置防火墙

firewall-cmd --permanent --add-port=2888/tcp
firewall-cmd --permanent --add-port=3888/tcp
firewall-cmd --permanent --add-port=2181/tcp
firewall-cmd --permanent --add-port=9092/tcp
firewall-cmd --permanent --add-port=2182/tcp
firewall-cmd --reload

1.4 创建存放zk数据的目录

mkdir -p /myhome/zk/data
mkdir -p /myhome/zk/datalog
mkdir -p /myhome/zk/conf/jaas

1.5 创建启动脚本~/zk-kafka/firststartzk.sh,内容如下:

#!/bin/bashcase ${myid} in
1)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=0.0.0.0:2888:3888;2181 server.2=192.168.132.12:2888:3888;2181 server.3=192.168.132.13:2888:3888;2181" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
zookeeper:3.6.1
;;
2)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=192.168.132.11:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=192.168.132.13:2888:3888;2181" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
zookeeper:3.6.1
;;
3)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=192.168.132.11:2888:3888;2181 server.2=192.168.132.12:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
zookeeper:3.6.1
;;
*)
echo "can not get myid value"
;;
esac

1.6 启动ZK集群,获取配置文件

cd ~/zk-kafka
./firststartzk.sh# 拷贝ZK配置文件
docker cp myzk_${IMAGE_NAME}:/conf /myhome/zk# 停止容器
docker stop myzk_${IMAGE_NAME}
docker rm myzk_${IMAGE_NAME}

1.7 修改配置文件/myhome/zk/conf/zoo.cfg

实现server-server 和 client-server认证

tee -a >>/myhome/zk/conf/zoo.cfg<<EOF# server-server authentication
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
quorum.auth.learner.saslLoginContext=QuorumLearner
quorum.auth.server.saslLoginContext=QuorumServer
quorum.cnxn.threads.size=6# client-server authentication
requireClientAuthScheme=sasl
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
EOF

1.8 创建JAAS配置文件/myhome/zk/conf/jaas/zk_server.conf,内容如下:

此文件中的Server用于客户端(比如zkClient.sh或kafka)连接，QuorumServer和QuorumLearner用于服务端之间的连接

tee -a >/myhome/zk/conf/jaas/zk_server.conf<<EOF
Server{org.apache.zookeeper.server.auth.DigestLoginModule requireduser_test="test123";
};QuorumServer {org.apache.zookeeper.server.auth.DigestLoginModule requireduser_test="test123";
};QuorumLearner {org.apache.zookeeper.server.auth.DigestLoginModule requiredusername="test"password="test123";
};
EOF

1.9 创建JAAS配置文件/myhome/zk/config/jaas/zk_client.conf,增加如下内容:

此文件用于客户端软件连接(比如zkClient.sh或kafka)

tee -a >/myhome/zk/config/jaas/zk_client.conf<<EOF
Client {org.apache.zookeeper.server.auth.DigestLoginModule requiredusername="test"password="test123";
};
EOF

1.10 创建正式启动ZK的脚本~/zk-kafka/startzk.sh

#!/bin/bashcase ${myid} in
1)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=0.0.0.0:2888:3888;2181 server.2=192.168.132.12:2888:3888;2181 server.3=192.168.132.13:2888:3888;2181" \
-e SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_server.conf" \
-e CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_client.conf" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
-v /myhome/zk/conf:/conf \
zookeeper:3.6.1
;;
2)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=192.168.132.11:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=192.168.132.13:2888:3888;2181" \
-e SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_server.conf" \
-e CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_client.conf" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
-v /myhome/zk/conf:/conf \
zookeeper:3.6.1
;;
3)
docker run -d --name=myzk_${IMAGE_NAME} \
--restart=always \
-p 2888:2888 \
-p 3888:3888 \
-p 2181:2181 \
-e ZOO_MY_ID=${myid} \
-e ZOO_SERVERS="server.1=192.168.132.11:2888:3888;2181 server.2=192.168.132.12:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181" \
-e SERVER_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_server.conf" \
-e CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/conf/jaas/zk_client.conf" \
-v /myhome/zk/data:/data \
-v /myhome/zk/datalog:/datalog \
-v /etc/localtime:/etc/localtime \
-v /myhome/zk/conf:/conf \
zookeeper:3.6.1
;;
*)
echo "can not get myid value"
;;
esac

1.11 启动ZK并查看日志

cd ~/zk-kafka
./startzk.sh# 查看日志
docker logs -f myzk_${IMAGE_NAME}
或
docker logs -f myzk_${IMAGE_NAME} |grep Sasl

输出类似如下:

2020-12-26 03:38:33,964 [myid:1] - INFO  [main:QuorumPeer@2463] - quorum.auth.enableSasl set to true
2020-12-26 03:38:33,964 [myid:1] - INFO  [main:QuorumPeer@2450] - quorum.auth.serverRequireSasl set to true
2020-12-26 03:38:33,964 [myid:1] - INFO  [main:QuorumPeer@2455] - quorum.auth.learnerRequireSasl set to true
2020-12-26 03:38:34,135 [myid:1] - INFO  [QuorumConnectionThread-[myid=1]-2:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test;  authorizationID=test.
2020-12-26 03:38:34,136 [myid:1] - INFO  [QuorumConnectionThread-[myid=1]-1:SaslQuorumAuthLearner@142] - Successfully completed the authentication using SASL. server addr: /192.168.132.12:3888, status: SUCCESS
2020-12-26 03:38:34,136 [myid:1] - INFO  [QuorumConnectionThread-[myid=1]-2:SaslQuorumAuthServer@110] - Successfully completed the authentication using SASL. learner addr: /192.168.132.12:55130
2020-12-26 03:38:34,138 [myid:1] - INFO  [QuorumConnectionThread-[myid=1]-3:SaslQuorumServerCallbackHandler@143] - Successfully authenticated learner: authenticationID=test;  authorizationID=test.
2020-12-26 03:38:34,138 [myid:1] - INFO  [QuorumConnectionThread-[myid=1]-3:SaslQuorumAuthServer@110] - Successfully completed the authentication using SASL. learner addr: /192.168.132.13:35982
2020-12-26 03:38:34,373 [myid:1] - INFO  [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):SaslQuorumAuthLearner@142] - Successfully completed the authentication using SASL. server addr: /192.168.132.13:2888, status: SUCCESS

1.12 进入容器

docker exec -it myzk_${IMAGE_NAME} /bin/bash# 执行客户端工具
bin/zkCli.sh

输出类似如下:

Welcome to ZooKeeper!
JLine support is enabled
2020-12-26 04:01:15,121 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):Login@320] - Client successfully logged in.
2020-12-26 04:01:15,126 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):SecurityUtils@70] - Client will use DIGEST-MD5 as SASL mechanism.
[zk: localhost:2181(CONNECTING) 0] 2020-12-26 04:01:15,173 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket connection to server localhost/127.0.0.1:2181.
2020-12-26 04:01:15,174 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config status: Will attempt to SASL-authenticate using Login Context section 'Client'
2020-12-26 04:01:15,182 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection established, initiating session, client: /127.0.0.1:40366, server: localhost/127.0.0.1:2181
2020-12-26 04:01:15,206 [myid:localhost:2181] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1420] - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x100003c85fe0000, negotiated timeout = 30000# 如果提示符出现connected字样,则说明连接服务端成功
[zk: localhost:2181(CONNECTED) 0]# 列出所有子节点
zk > ls -R /

到此为止，实现了ZK的server-server 和 client-server的认证

2. kafka

2.1 创建存放数据的目录

mkdir -p /myhome/kafka/data
mkdir -p /myhome/kafka/logs
mkdir -p /myhome/kafka/jaas

2.2 创建JAAS配置文件/myhome/kafka/jaas/zk_client.conf,增加如下内容:

tee -a >>/myhome/kafka/jaas/zk_client.conf<<EOF
Client {org.apache.zookeeper.server.auth.DigestLoginModule requiredusername="test"password="test123";
};
EOF

2.3 启动kafka

docker run -d --name=mykafka_${IMAGE_NAME} \
--restart=always \
-p 9092:9092 \
-v /etc/localtime:/etc/localtime \
-v /myhome/kafka/data:/kafka \
-v /myhome/kafka/logs:/opt/kafka/logs \
-v /myhome/kafka/jaas:/mnt \
-e KAFKA_ADVERTISED_HOST_NAME=${MYIP} \
-e HOST_IP=${MYIP} \
-e KAFKA_ADVERTISED_PORT=9092 \
-e KAFKA_ZOOKEEPER_CONNECT=192.168.132.11:2181,192.168.132.12:2181,192.168.132.13:2181 \
-e KAFKA_BROKER_ID=${myid} \
-e KAFKA_OPTS="-Djava.security.auth.login.config=/mnt/zk_client.conf" \
-e KAFKA_ZOOKEEPER_SET_ACT=true \
wurstmeister/kafka:2.13-2.6.0

2.4 查看日志

docker logs -f mykafka_${IMAGE_NAME}

2.5 验证kafka

# 进入容器
docker exec -it mykafka_${IMAGE_NAME} /bin/bashcd /opt/kafka# 创建topic
bin/kafka-topics.sh --create --zookeeper ${MYIP}:2181 --replication-factor 1 --partitions 1 --topic test# 查看Topic
bin/kafka-topics.sh --list --zookeeper ${MYIP}:2181# 生产者：向Kafka发送消息
bin/kafka-console-producer.sh --broker-list 0.0.0.0:9092 --topic test# 消费者: 消费刚刚发送的消息
bin/kafka-console-consumer.sh --bootstrap-server 0.0.0.0:9092 --topic test --from-beginning

2.6 验证ZK

# 进入容器
docker exec -it myzk_${IMAGE_NAME} /bin/bash# 执行客户端工具
bin/zkCli.sh# 列出所有子节点
zk> ls -R /# 查看之前创建的主题:test的ACL
zk> getAcl /brokers/topics/test

输出如下:

# 表示采用了sasl认证，用户名是test,权限是cdrwa(即全部权限)
'sasl,'test
: cdrwa
# 表示任意人均可访问,权限是r(即只读)
'world,'anyone
: r

ZK实现SASL认证+Kafka连接ZK相关推荐

1. kafka与zk的关系（一）

   一个典型的kafka集群中包含若干个Producer,若干个broker(一般broker数量越多,集群吞吐率越高),若干Consumer Group,以及一个zk集群,kafka通过zk管理集群配置 ...

2. Kafka与zk的关系及连接参数bootstrap.server的正确理解

   1.我们知道旧版Kafka,用的是zookeeper地址而非bootstrap.servers, 那么新版 kafka 消费者.生产者配置为何使用 bootstrap-servers 而不是 zook ...

3. zookeeper和kafka的SASL认证以及生产实践

   一.什么是zookeeper? ZooKeeper是一个集中的服务,用于维护配置信息.命名.提供分布式同步以及提供组服务.所有这些类型的服务都以某种形式被分布式应用程序使用.每次它们被实现时,都有大量 ...

4. kafka SASL认证介绍及自定义SASL PLAIN认证功能

   文章目录 kafka 2.x用户认证方式小结 SASL/PLAIN实例(配置及客户端) broker配置 客户端配置 自定义SASL/PLAIN认证(二次开发) kafka2新的callback接口介 ...

5. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（九）安装kafka_2.11-1.1.0

   如何搭建配置centos虚拟机请参考<Kafka:ZK+Kafka+Spark Streaming集群环境搭建(一)VMW安装四台CentOS,并实现本机与它们能交互,虚拟机内部实现可以上网.& ...

6. 大数据处理工具Kafka、Zk、Spark

   搭建kafka和zk集群环境 安装环境 MAC操作系统VMware Fusion虚拟机3个centos7服务器 安装虚拟机 飞机票 安装虚拟机Centos系统并安装Docker过程记录 安装包下载 h ...

7. Kafka：ZK+Kafka+Spark Streaming集群环境搭建（二十一）NIFI1.7.1安装

   一.nifi基本配置 1. 修改各节点主机名,修改/etc/hosts文件内容. 192.168.0.120master192.168.0.121slave1192.168.0.122 slave2 ...

8. java本地连接zk集群

   [README]基于centos8 搭建zookeeper 集群,refer2 https://blog.csdn.net/PacosonSWJTU/article/details/111409079 ...

9. 10-10-020-简介-kafka的ZK上面的存储

   文章目录 1.视界 1. kafka在ZK上都存储了什么呢? 2. ZkCli.sh命令下如图 3. 节点表格 3.节点信息 3.1 kafka根节点信息 3.2 topic注册信息 3.3 part ...

最新文章

1. MATLAB句柄相关函数和代码示例（设置美化图表属性）
2. 2020年旷世校招JAVA岗笔试第二题
3. 前端要凉？微软开源Sketch2Code，草图秒变代码
4. Document is invalid: no grammar found. at (null:3:8)
5. java中容易被忽视的基本概念
6. java 路由_RabbitMQ入门：路由(Routing)
7. C++之一个函数链的简单例子（分文件实现）
8. [Contest20180418]物理竞赛
9. [debug] RuntimeError: “nll_loss_forward_reduce_cuda_kernel_2d_index“ not implemented for ‘Int‘
10. Chrome插件之一键保存网页为PDF1.1发布
11. Node.js入门初体验
12. BZOJ1079[SCOI2008] 着色方案
13. Angr安装与使用之使用篇（六）
14. MATLAB 2018a安装
15. 人型自走输入法（网页日语输入法）
16. IOS 关于扬声器和听话筒的设置 ----自己试验过的，可以达到扩音器和听筒播放的效果...
17. 路由器的基本配置和直连路由
18. 我们无法设置移动热点_支付宝微信设置了密码，为何超市扫码枪能将钱扫走？多数人不了解...
19. Windows XP的图标结构
20. 积木报表VS睿思BI报表

热门文章

1. 伏临扰雨（北京的雨季）
2. 【BP神经网络】西瓜分类和鸢尾花分类
3. python overflowerror_Python中的OverflowError:（34，'结果太大'）
4. 即时通讯sdk哪一个好？
5. 《庄子·内篇·逍遥游第一》
6. 片上总线Wishbone 学习（十）总线周期之单写读操作
7. 在Python中用WordCloud生成聊天记录热点词汇词云图
8. d3 地图长宽比调整
9. 借鉴华为HiLink实现微信小程序智能配网功能
10. 学计算机颈椎痛,近九成电脑族曾颈椎疼痛，怎么办？