Kafka服务端脚本详解(2)一log,verifiable
2019独角兽企业重金招聘Python工程师标准>>>
脚本名称 | 脚本用途 |
---|---|
kafka-log-dirs.sh | 查看指定broker上日志目录使用情况 |
kafka-verifiable-consumer.sh | 检验kafka消费者 |
kafka-verifiable-producer.sh | 检验kafka生产者 |
kafka-log-dirs.sh
--bootstrap-server
kafka地址
--broker-list
要查询的broker地址列表,broker之间逗号隔开,不配置该命令则查询所有broker
--topic-list
指定查询的topic列表,逗号隔开
--command-config
配置Admin Client
--describe
显示详情
1[root@10 kafka_2.11-2.2.0]# bin/kafka-log-dirs.sh --bootstrap-server 10.211.55.3:9092 --describe --broker-list 0 --topic-list first,topic-3
2Querying brokers for log directories information
3Received log directory information from brokers 0
4{"version":1,"brokers":[{"broker":0,"logDirs":[{"logDir":"/tmp/kafka-logs","error":null,"partitions":[{"partition":"topic-3-0","size":474,"offsetLag":0,"isFuture":false},{"partition":"first-0","size":310,"offsetLag":0,"isFuture":false}]}]}]}
kafka-verifiable-consumer.sh
--broker-list
broker列表, HOST1:PORT1,HOST2:PORT2,…
--topic
要消费的topic
--group-id
消费组id
--max-messages
最大消费消息数量,默认-1,一直消费
1#设置消费两次后,自动停止2[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo --max-messages 23{"timestamp":1558869583036,"name":"startup_complete"}4{"timestamp":1558869583329,"name":"partitions_revoked","partitions":[]}5{"timestamp":1558869583366,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}6{"timestamp":1558869590352,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":37,"maxOffset":37}]}7{"timestamp":1558869590366,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":38}],"success":true}8{"timestamp":1558869595328,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":38,"maxOffset":38}]}9{"timestamp":1558869595335,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":39}],"success":true}
10{"timestamp":1558869595355,"name":"shutdown_complete"}
--session-timeout
消费者会话超时时间,默认30000ms,服务端如果在该时间内没有接收到消费者的心跳,就会将该消费者从消费组中删除
--enable-autocommit
自动提交,默认false
1#比较一下两者的差别2#没有--enable-autocommit3[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo4{"timestamp":1558875063613,"name":"startup_complete"}5{"timestamp":1558875063922,"name":"partitions_revoked","partitions":[]}6{"timestamp":1558875063952,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}7{"timestamp":1558875069603,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":47,"maxOffset":47}]}8{"timestamp":1558875069614,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":48}],"success":true}9
10#有--enable-autocommit
11[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo --enable-autocommit
12{"timestamp":1558874772119,"name":"startup_complete"}
13{"timestamp":1558874772408,"name":"partitions_revoked","partitions":[]}
14{"timestamp":1558874772449,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}
15{"timestamp":1558874820898,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":46,"maxOffset":46}]}
--reset-policy
设置消费偏移量,earliest从头开始消费,latest从最近的开始消费,none抛出异常,默认earliest
--assignment-strategy
消费者的分区配置策略, 默认 RangeAssignor
--consumer.config
配置文件
kafka-verifiable-producer.sh
该脚本可以生产测试数据发送到指定topic,并将数据已json格式打印到控制台
--topic
主题名称
--broker-list
broker列表, HOST1:PORT1,HOST2:PORT2,…
--max-messages
最大消息数量,默认-1,一直生产消息
--throughput
设置吞吐量,默认-1
--acks
指定分区中必须有多少个副本收到这条消息,才算消息发送成功,默认-1
--producer.config
配置文件
--message-create-time
设置消息创建的时间,时间戳
--value-prefix
设置消息前缀
--repeating-keys
key从0开始,每次递增1,直到指定的值,然后再从0开始
1[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-producer.sh --broker-list 10.211.55.3:9092 --topic first --message-create-time 1527351382000 --value-prefix 1 --repeating-keys 10 --max-messages 202{"timestamp":1558877565069,"name":"startup_complete"}3{"timestamp":1558877565231,"name":"producer_send_success","key":"0","value":"1.0","topic":"first","partition":0,"offset":1541118}4{"timestamp":1558877565238,"name":"producer_send_success","key":"1","value":"1.1","topic":"first","partition":0,"offset":1541119}5{"timestamp":1558877565238,"name":"producer_send_success","key":"2","value":"1.2","topic":"first","partition":0,"offset":1541120}6{"timestamp":1558877565238,"name":"producer_send_success","key":"3","value":"1.3","topic":"first","partition":0,"offset":1541121}7{"timestamp":1558877565238,"name":"producer_send_success","key":"4","value":"1.4","topic":"first","partition":0,"offset":1541122}8{"timestamp":1558877565239,"name":"producer_send_success","key":"5","value":"1.5","topic":"first","partition":0,"offset":1541123}9{"timestamp":1558877565239,"name":"producer_send_success","key":"6","value":"1.6","topic":"first","partition":0,"offset":1541124}
10{"timestamp":1558877565239,"name":"producer_send_success","key":"7","value":"1.7","topic":"first","partition":0,"offset":1541125}
11{"timestamp":1558877565239,"name":"producer_send_success","key":"8","value":"1.8","topic":"first","partition":0,"offset":1541126}
12{"timestamp":1558877565239,"name":"producer_send_success","key":"9","value":"1.9","topic":"first","partition":0,"offset":1541127}
13{"timestamp":1558877565239,"name":"producer_send_success","key":"0","value":"1.10","topic":"first","partition":0,"offset":1541128}
14{"timestamp":1558877565239,"name":"producer_send_success","key":"1","value":"1.11","topic":"first","partition":0,"offset":1541129}
15{"timestamp":1558877565239,"name":"producer_send_success","key":"2","value":"1.12","topic":"first","partition":0,"offset":1541130}
16{"timestamp":1558877565240,"name":"producer_send_success","key":"3","value":"1.13","topic":"first","partition":0,"offset":1541131}
17{"timestamp":1558877565240,"name":"producer_send_success","key":"4","value":"1.14","topic":"first","partition":0,"offset":1541132}
18{"timestamp":1558877565241,"name":"producer_send_success","key":"5","value":"1.15","topic":"first","partition":0,"offset":1541133}
19{"timestamp":1558877565244,"name":"producer_send_success","key":"6","value":"1.16","topic":"first","partition":0,"offset":1541134}
20{"timestamp":1558877565244,"name":"producer_send_success","key":"7","value":"1.17","topic":"first","partition":0,"offset":1541135}
21{"timestamp":1558877565244,"name":"producer_send_success","key":"8","value":"1.18","topic":"first","partition":0,"offset":1541136}
22{"timestamp":1558877565244,"name":"producer_send_success","key":"9","value":"1.19","topic":"first","partition":0,"offset":1541137}
23{"timestamp":1558877565262,"name":"shutdown_complete"}
24{"timestamp":1558877565263,"name":"tool_data","sent":20,"acked":20,"target_throughput":-1,"avg_throughput":100.50251256281408}
Reference:
[1] [Kafka之实战指南-朱小厮] http://wxpic.178le.net/FhlwV5J7M9LbXElyMGQMPE67CzGr
[2] [阿飞的博客] https://blog.csdn.net/feelwing1314
[3] [Apache Kafka] http://kafka.apache.org/documentation/
转载于:https://my.oschina.net/itsaysay/blog/3054319
Kafka服务端脚本详解(2)一log,verifiable相关推荐
- Nuxt SSR 服务端渲染 详解
Nuxt SSR 服务端渲染 详解 1.Nuxt项目构建 2.Nuxt的生命周期 2.1 nuxtServerInit 钩子 2.2 middleware 中间件 2.3 validate 数据校验 ...
- NodeJS+Express+mySQL服务端开发详解
NodeJS+Express+mySQL服务端开发详解 随着NodeJS的发展,现在已经被很多人熟知,NodeJS已经成为了前端开发人员必备的技能.本文不会对NodeJS过多介绍 如果你感兴趣可以访问 ...
- 我的世界服务器怎么找到指定路径,[小白]MC服务端目录详解
服务器的前期准备工作都做好了,是时候上传服务端到服务器了.不过--先等等,在上传前你应该先了解一下服务端目录的结构以便于后期的维护! 如图所示的是MCPC+1.6.4的服务端结构示意图,其他开服端基本 ...
- 使用KBEngine开发UE4服务端——开发详解
使用KBEngine开发UE4服务端--开发详解 1. 简述 开始正式进入开发,网络上关于开发和修改文件的文章较少,这里会记录自己开发过程中需要更改的地方和一些流程. 2. 前期的配置说明 根据前文配 ...
- QT中使用C++ socket通信,socket通信原理三次握手和四次握手详解、客户端与服务端实例详解
对TCP/IP.UDP.Socket编程这些词你不会很陌生吧?随着网络技术的发展,这些词充斥着我们的耳朵.那么我想问: 1. 什么是TCP/IP.UDP? 2. Soc ...
- mysql5.7只安装服务端_Windows Mysql5.7.11 服务端安装详解
MySQL服务端安装过程主要是选择安装类型(自定义.典型和完整版),一般我都会选择自定义主要是为了安装到指定目录上,此过程省略,下面着重记录配置过程. 1.配置my.ini文件 在解压的目录下面复制m ...
- 服务端渲染详解(SSR)
综述:知识扩展,深化认识 1.服务端渲染产生背景介绍 自从SPA(单页面应用)诞生后,其顺滑的用户体验让大家趋之若鹜,但由此也产生了两个问题: 由于其是在浏览器端完成了页面的路由控制,不利于网站的SE ...
- teleport最新版不支持mysql_Teleport服务端配置详解
服务端配置文件说明 本文中%INSTDIR%指teleport服务端安装目录,默认位于/usr/local/teleport/. teleport配置文件位于%INSTDIR%/data/etc目录下 ...
- 千年服务器角色信息在哪个文件,千年服务端文件详解
在gate1000下的addr.txt文件里添加上你服务器的地址和名字,例如:192.168.0.1,3053,XXXXX,XXX ,在目录下的gate.ini里修改 "SERVERNAME ...
最新文章
- ElasticSearch之Java Api 测试
- CodeForces-1294B排序+pair使用
- UU看书于今日成功上线,各大小说网站发来贺电!!
- 将excel文件中的数据导入到mysql
- C#异步编程看这篇就够了
- [剑指offer][JAVA]面试题第[34]题[二叉树中和为某一值的路径][回溯]
- 奇怪,勒索软件Ryuk 新版本把这些 *UNIX 文件夹列入了黑名单
- spring包装hibernate_全网最全Spring系列面试题129道(附答案解析)
- 多物理场面向对象模拟环境MOOSE学习手册
- request.setAttribute()怎么用的?
- signature=29f9d891eda46899a29591e507a569b3,NEW MARKER OF BREAST TUMORS FROM THE LUMINAL-B SYBTYPE
- 多种方式99.9%解决从PDF复制文字后乱码问题
- python使用gps设备
- graphpad数据小数点_Image J和Graphpad如何对Western Blot条带灰度分析【干货】 | 每日生物评论...
- 线性代数:第五章 相似矩阵及二次型(2)二次型及其标准型 配平方法 合同变换法 正定二次型
- i7 10875h和i7 9750h对比差距大吗
- linux命令行下的tab键作用
- 臀部大的美女最令男人着迷
- Power BI中如何处理相同名称的客户
- 2022年十一届认证杯C题