kafka python client：PyKafka vs kafka-python

引用：https://github.com/Parsely/pykafka/issues/334

@emmett9001写道

@microamp Thanks, this is a great idea. There's currently no documentation on this, but to my knowledge the main differences are the specifics of the Python API and PyKafka's implementation of theBalancedConsumer. PyKafka strives to keep the API as pythonic as possible, which means using useful features of the language where appropriate for client code simplicity. This includes things like context managers for object cleanup and futures for asynchronous error handling. PyKafka's balanced consumer implements the Kafka project's notion of the "high level consumer", which uses ZooKeeper to balance consumption of partitions between multiple nodes in a consumer group. From what I understand, kafka-python is waiting until Kafka 0.9, when this functionality will be supported natively by the Kafka server itself, to implement self-balancing consumers.
Also, the last time we did a speed test (which was admittedly a while ago at this point), PyKafka's consumer outperformed kafka-python. I unfortunately no longer have the results from that test, so you may not want to bet too hard on PyKafka being significantly faster or slower - just figured I'd mention it.

注：

1、PyKafka 尽量保持了 python 的接口方式，包括上下文处理和异步的异常处理
2、PyKafka 支持 ZooKeeper 做消费者负载均衡。而 kafka-python 直到 Kafka 0.9 才支持消费者负载均衡，并且是依靠kafka服务实现的。

@emmett9001写道

Some more research - there are differences in the versions of python supported by each library. PyKafka supports 2.7, 3.4, 3.5, and pypy, while kafka-python adds 2.6 and removes 3.5 support. kafka-python also requires a ZooKeeper connection for offset management, which PyKafka does not. kafka-python supports versions of Kafka from 0.8.0 to 0.8.2, where PyKafka only supports 0.8.2.

注：

1、Python 版本：kafka-python 支持 2.6，不支持 3.5， PyKafak 支持 2.7，3.4，3.5

2、Kafka 版本：kafka-python 支持 0.8.0 ~ 0.8.2，PyKafka 只支持 0.8.2

@ottomata写道

A difference between kafka-python and pykafka is the producer interface. kafka-python does not require that you know the topic when instantiating the producer. This is convenient if you need to produce to topics dynamically based on input (which I do!) :)

注：

kafka-python 在初始化 producer 时，不需要知道 topic。（言外之意：PyKafka 需要？）

@cscheffler写道

@emmett9001 @ottomata Just got pointed at this thread and thought I'd make a late contribution.

We compared pykafka and kafka-python about 2 months ago while trying to decide which one to use. In the end, the deciding factor for us was that balanced consumers were much easier to manage in pykafka.

Also, we discovered later, a pykafka producer doesn't die on Kafka broker restart, while our kafka-python producers did.

Below are performance figures from a 3-node Kafka cluster running in EC2, using a single producer or consumer. The three numbers for each test are the quartiles measured for the test.

pykafka producer: 41400 – 46500 – 50200 messages per second
pykafka consumer: 12100 – 14400 – 23700 messages per second
kafka-python producer: 26500 – 27700 – 29500 messages per second
kafka-python consumer: 35000 – 37300 – 39100 messages per second
So, for clarification, the median performance of a pykafka producer was 46500 messages per second, with a quartile range of 41400 (25th percentile) to 50200 (75th percentile). Hope that makes sense.

注：
1、当 broker 重启后，kafka-python producer 会死掉，而 PyKafka 不会

2、作为生产者，PyKafka 性能更好；作为消费者，kafka-python 性能更好

kafka python client：PyKafka vs kafka-python相关推荐

Kafka系列之：深入理解Kafka 主题、分区、副本、LEO、ISR、HW、Kafka的主写主读和分区leader选举
Kafka系列之:深入理解Kafka 主题.分区.副本.LEO.ISR.HW.Kafka的主写主读和分区leader选举一.Kafka重要知识点提炼二.详细介绍Kafka 主题.分区.副本.LEO ...
Kafka系列之：不重启kafka集群设置kafka topic数据保留时间
Kafka系列之:不重启kafka集群设置kafka topic数据保留时间一.kafka topic数据保留3天的bash命令二.查看kafka删除数据日志三.批量设置上千个topic保留3天 ...
linux运行脚本文件python,Python脚本：Linux自动化执行Python脚本
1.环境及其工具: ubuntu 16.04 python2.7(自带) pip2.7(安装) virtualenv(安装) crontab (自带) 2.pip2.7安装 (1)尝试使用 sudo ...
Python编程：twine模块打包python项目上传pypi
注册账号(重要) https://pypi.org 可以配置到$HOME/.pypirc文件中,就不用多次输入了 [pypi] username = <username> password ...
Python疑难杂症：SyntaxError: Non-ASCII character Python中文处理问题
python的中文问题一直是困扰新手的头疼问题,这篇文章将给你详细地讲解一下这方面的知识.当然,几乎可以确定的是,在将来的版本中,python会彻底解决此问题,不用我们这么麻烦了. 先来看看pytho ...
五分钟学会python编程_每天五分钟python编程：生成器技术是python语言最强大的技术之一...
动态方式生成列表这样的方式生成列表会有一个特点:就是列表中的元素都是一下子生成的,因为这里之生成10个元素,因为元素数量比较少,所以体会不到,当我们这样的话:[i for i in range(10 ...
kafka消费报错：org.apache.kafka.common.errors.WakeupException: null
文章目录 1. 背景 1. 背景我在测试类中写一个kafka消费者的线程,大概结构如下 @Testpublic void startMeteorTask() {thread.run(){
python随笔：用pyinstaller 将python代码打包成exe执行文件
1.前言经过几个星期的努力,终于完成了一个PyQt5项目.但日常使用或给别人使用代码模式非常不便,所以就产生了把代码打包成exe执行文件的想法. 2.pyinstaller 安装在命令行执行以下语 ...
第1节 kafka消息队列：7、kafka的消费模型
转载于:https://www.cnblogs.com/mediocreWorld/p/11220511.html

kafka python client：PyKafka vs kafka-python

引用：https://github.com/Parsely/pykafka/issues/334

kafka python client：PyKafka vs kafka-python相关推荐

最新文章

热门文章