2019独角兽企业重金招聘Python工程师标准>>>

脚本名称 脚本用途
kafka-log-dirs.sh 查看指定broker上日志目录使用情况
kafka-verifiable-consumer.sh 检验kafka消费者
kafka-verifiable-producer.sh 检验kafka生产者

kafka-log-dirs.sh

--bootstrap-server

kafka地址

--broker-list

要查询的broker地址列表,broker之间逗号隔开,不配置该命令则查询所有broker

--topic-list

指定查询的topic列表,逗号隔开

--command-config

配置Admin Client

--describe

显示详情

1[root@10 kafka_2.11-2.2.0]# bin/kafka-log-dirs.sh --bootstrap-server 10.211.55.3:9092 --describe --broker-list 0 --topic-list first,topic-3
2Querying brokers for log directories information
3Received log directory information from brokers 0
4{"version":1,"brokers":[{"broker":0,"logDirs":[{"logDir":"/tmp/kafka-logs","error":null,"partitions":[{"partition":"topic-3-0","size":474,"offsetLag":0,"isFuture":false},{"partition":"first-0","size":310,"offsetLag":0,"isFuture":false}]}]}]}

kafka-verifiable-consumer.sh

--broker-list

broker列表, HOST1:PORT1,HOST2:PORT2,…

--topic

要消费的topic

--group-id

消费组id

--max-messages

最大消费消息数量,默认-1,一直消费

 1#设置消费两次后,自动停止2[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo --max-messages 23{"timestamp":1558869583036,"name":"startup_complete"}4{"timestamp":1558869583329,"name":"partitions_revoked","partitions":[]}5{"timestamp":1558869583366,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}6{"timestamp":1558869590352,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":37,"maxOffset":37}]}7{"timestamp":1558869590366,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":38}],"success":true}8{"timestamp":1558869595328,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":38,"maxOffset":38}]}9{"timestamp":1558869595335,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":39}],"success":true}
10{"timestamp":1558869595355,"name":"shutdown_complete"}

--session-timeout

消费者会话超时时间,默认30000ms,服务端如果在该时间内没有接收到消费者的心跳,就会将该消费者从消费组中删除

--enable-autocommit

自动提交,默认false

 1#比较一下两者的差别2#没有--enable-autocommit3[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo4{"timestamp":1558875063613,"name":"startup_complete"}5{"timestamp":1558875063922,"name":"partitions_revoked","partitions":[]}6{"timestamp":1558875063952,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}7{"timestamp":1558875069603,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":47,"maxOffset":47}]}8{"timestamp":1558875069614,"name":"offsets_committed","offsets":[{"topic":"first","partition":0,"offset":48}],"success":true}9
10#有--enable-autocommit
11[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-consumer.sh --broker-list 10.211.55.3:9092 --topic first --group-id group.demo --enable-autocommit
12{"timestamp":1558874772119,"name":"startup_complete"}
13{"timestamp":1558874772408,"name":"partitions_revoked","partitions":[]}
14{"timestamp":1558874772449,"name":"partitions_assigned","partitions":[{"topic":"first","partition":0}]}
15{"timestamp":1558874820898,"name":"records_consumed","count":1,"partitions":[{"topic":"first","partition":0,"count":1,"minOffset":46,"maxOffset":46}]}

--reset-policy

设置消费偏移量,earliest从头开始消费,latest从最近的开始消费,none抛出异常,默认earliest

--assignment-strategy

消费者的分区配置策略, 默认 RangeAssignor

--consumer.config

配置文件


kafka-verifiable-producer.sh

该脚本可以生产测试数据发送到指定topic,并将数据已json格式打印到控制台

--topic

主题名称

--broker-list

broker列表, HOST1:PORT1,HOST2:PORT2,…

--max-messages

最大消息数量,默认-1,一直生产消息

--throughput

设置吞吐量,默认-1

--acks

指定分区中必须有多少个副本收到这条消息,才算消息发送成功,默认-1

--producer.config

配置文件

--message-create-time

设置消息创建的时间,时间戳

--value-prefix

设置消息前缀

--repeating-keys

key从0开始,每次递增1,直到指定的值,然后再从0开始

 1[root@10 kafka_2.11-2.2.0]# bin/kafka-verifiable-producer.sh --broker-list 10.211.55.3:9092 --topic first --message-create-time 1527351382000 --value-prefix 1 --repeating-keys 10 --max-messages 202{"timestamp":1558877565069,"name":"startup_complete"}3{"timestamp":1558877565231,"name":"producer_send_success","key":"0","value":"1.0","topic":"first","partition":0,"offset":1541118}4{"timestamp":1558877565238,"name":"producer_send_success","key":"1","value":"1.1","topic":"first","partition":0,"offset":1541119}5{"timestamp":1558877565238,"name":"producer_send_success","key":"2","value":"1.2","topic":"first","partition":0,"offset":1541120}6{"timestamp":1558877565238,"name":"producer_send_success","key":"3","value":"1.3","topic":"first","partition":0,"offset":1541121}7{"timestamp":1558877565238,"name":"producer_send_success","key":"4","value":"1.4","topic":"first","partition":0,"offset":1541122}8{"timestamp":1558877565239,"name":"producer_send_success","key":"5","value":"1.5","topic":"first","partition":0,"offset":1541123}9{"timestamp":1558877565239,"name":"producer_send_success","key":"6","value":"1.6","topic":"first","partition":0,"offset":1541124}
10{"timestamp":1558877565239,"name":"producer_send_success","key":"7","value":"1.7","topic":"first","partition":0,"offset":1541125}
11{"timestamp":1558877565239,"name":"producer_send_success","key":"8","value":"1.8","topic":"first","partition":0,"offset":1541126}
12{"timestamp":1558877565239,"name":"producer_send_success","key":"9","value":"1.9","topic":"first","partition":0,"offset":1541127}
13{"timestamp":1558877565239,"name":"producer_send_success","key":"0","value":"1.10","topic":"first","partition":0,"offset":1541128}
14{"timestamp":1558877565239,"name":"producer_send_success","key":"1","value":"1.11","topic":"first","partition":0,"offset":1541129}
15{"timestamp":1558877565239,"name":"producer_send_success","key":"2","value":"1.12","topic":"first","partition":0,"offset":1541130}
16{"timestamp":1558877565240,"name":"producer_send_success","key":"3","value":"1.13","topic":"first","partition":0,"offset":1541131}
17{"timestamp":1558877565240,"name":"producer_send_success","key":"4","value":"1.14","topic":"first","partition":0,"offset":1541132}
18{"timestamp":1558877565241,"name":"producer_send_success","key":"5","value":"1.15","topic":"first","partition":0,"offset":1541133}
19{"timestamp":1558877565244,"name":"producer_send_success","key":"6","value":"1.16","topic":"first","partition":0,"offset":1541134}
20{"timestamp":1558877565244,"name":"producer_send_success","key":"7","value":"1.17","topic":"first","partition":0,"offset":1541135}
21{"timestamp":1558877565244,"name":"producer_send_success","key":"8","value":"1.18","topic":"first","partition":0,"offset":1541136}
22{"timestamp":1558877565244,"name":"producer_send_success","key":"9","value":"1.19","topic":"first","partition":0,"offset":1541137}
23{"timestamp":1558877565262,"name":"shutdown_complete"}
24{"timestamp":1558877565263,"name":"tool_data","sent":20,"acked":20,"target_throughput":-1,"avg_throughput":100.50251256281408}

Reference: 
[1] [Kafka之实战指南-朱小厮] http://wxpic.178le.net/FhlwV5J7M9LbXElyMGQMPE67CzGr 
[2] [阿飞的博客] https://blog.csdn.net/feelwing1314 
[3] [Apache Kafka] http://kafka.apache.org/documentation/

转载于:https://my.oschina.net/itsaysay/blog/3054319

Kafka服务端脚本详解(2)一log,verifiable相关推荐

  1. Nuxt SSR 服务端渲染 详解

    Nuxt SSR 服务端渲染 详解 1.Nuxt项目构建 2.Nuxt的生命周期 2.1 nuxtServerInit 钩子 2.2 middleware 中间件 2.3 validate 数据校验 ...

  2. NodeJS+Express+mySQL服务端开发详解

    NodeJS+Express+mySQL服务端开发详解 随着NodeJS的发展,现在已经被很多人熟知,NodeJS已经成为了前端开发人员必备的技能.本文不会对NodeJS过多介绍 如果你感兴趣可以访问 ...

  3. 我的世界服务器怎么找到指定路径,[小白]MC服务端目录详解

    服务器的前期准备工作都做好了,是时候上传服务端到服务器了.不过--先等等,在上传前你应该先了解一下服务端目录的结构以便于后期的维护! 如图所示的是MCPC+1.6.4的服务端结构示意图,其他开服端基本 ...

  4. 使用KBEngine开发UE4服务端——开发详解

    使用KBEngine开发UE4服务端--开发详解 1. 简述 开始正式进入开发,网络上关于开发和修改文件的文章较少,这里会记录自己开发过程中需要更改的地方和一些流程. 2. 前期的配置说明 根据前文配 ...

  5. QT中使用C++ socket通信,socket通信原理三次握手和四次握手详解、客户端与服务端实例详解

    对TCP/IP.UDP.Socket编程这些词你不会很陌生吧?随着网络技术的发展,这些词充斥着我们的耳朵.那么我想问: 1.         什么是TCP/IP.UDP? 2.         Soc ...

  6. mysql5.7只安装服务端_Windows Mysql5.7.11 服务端安装详解

    MySQL服务端安装过程主要是选择安装类型(自定义.典型和完整版),一般我都会选择自定义主要是为了安装到指定目录上,此过程省略,下面着重记录配置过程. 1.配置my.ini文件 在解压的目录下面复制m ...

  7. 服务端渲染详解(SSR)

    综述:知识扩展,深化认识 1.服务端渲染产生背景介绍 自从SPA(单页面应用)诞生后,其顺滑的用户体验让大家趋之若鹜,但由此也产生了两个问题: 由于其是在浏览器端完成了页面的路由控制,不利于网站的SE ...

  8. teleport最新版不支持mysql_Teleport服务端配置详解

    服务端配置文件说明 本文中%INSTDIR%指teleport服务端安装目录,默认位于/usr/local/teleport/. teleport配置文件位于%INSTDIR%/data/etc目录下 ...

  9. 千年服务器角色信息在哪个文件,千年服务端文件详解

    在gate1000下的addr.txt文件里添加上你服务器的地址和名字,例如:192.168.0.1,3053,XXXXX,XXX ,在目录下的gate.ini里修改 "SERVERNAME ...

最新文章

  1. ElasticSearch之Java Api 测试
  2. CodeForces-1294B排序+pair使用
  3. UU看书于今日成功上线,各大小说网站发来贺电!!
  4. 将excel文件中的数据导入到mysql
  5. C#异步编程看这篇就够了
  6. [剑指offer][JAVA]面试题第[34]题[二叉树中和为某一值的路径][回溯]
  7. 奇怪,勒索软件Ryuk 新版本把这些 *UNIX 文件夹列入了黑名单
  8. spring包装hibernate_全网最全Spring系列面试题129道(附答案解析)
  9. 多物理场面向对象模拟环境MOOSE学习手册
  10. request.setAttribute()怎么用的?
  11. signature=29f9d891eda46899a29591e507a569b3,NEW MARKER OF BREAST TUMORS FROM THE LUMINAL-B SYBTYPE
  12. 多种方式99.9%解决从PDF复制文字后乱码问题
  13. python使用gps设备
  14. graphpad数据小数点_Image J和Graphpad如何对Western Blot条带灰度分析【干货】 | 每日生物评论...
  15. 线性代数:第五章 相似矩阵及二次型(2)二次型及其标准型 配平方法 合同变换法 正定二次型
  16. i7 10875h和i7 9750h对比差距大吗
  17. linux命令行下的tab键作用
  18. 臀部大的美女最令男人着迷
  19. Power BI中如何处理相同名称的客户
  20. 2022年十一届认证杯C题

热门文章

  1. 记录自己又一次的熬夜刷剧后感想
  2. bss、data、rodata和text
  3. STM32L431RCT6单片机自制板复位按键不灵的可能问题
  4. VMware设置虚拟机的网络为桥接模式,虚拟机使用主机的网段,虚拟机设置和主机一样的网关(网段)
  5. QoS服务质量五QoS边界行为之流量整形
  6. 【Lua进阶系列】lua_Stack
  7. 纯css3制作写轮眼开眼及进化过程
  8. 网络爬虫day1:python中的request模块基本使用
  9. win10内存占用高达四五十的解决方法
  10. excel删除无尽空白行_Excel数据联动word整理技巧,双向复制粘贴,快捷转换