关于 Apache Pulsar

Apache Pulsar 是 Apache 软件基金会顶级项目，是下一代云原生分布式消息流平台，集消息、存储、轻量化函数式计算为一体，采用计算与存储分离架构设计，支持多租户、持久化存储、多机房跨区域数据复制，具有强一致性、高吞吐、低延时及高可扩展性等流数据存储特性。
GitHub 地址：http://github.com/apache/pulsar/

导语

各位小伙伴们，2021 年 6 月 Pulsar 社区月报来啦！

感谢以下各位社区成员本月对 Pulsar 项目的支持，让 Apache Pulsar 继续发光发热！（排名不分先后，看看你有没有上榜 ????????）：

eolivelli、ciaocloud、dlg99、fantapsody、wolfstudy、315157973、codelipenghui、sursingh、jerrypeng、sijie、aahmed-se、linlinnn、RobertIndie、wangjialing218、wuzhanpeng、dockerzhang、ru1996、lzhpo、michaeljmarshall、newur、rdhabalia、lhotari、merlimat、aloyszhang、congbobo184、lbenc135、BewareMyPower、vmalloc、Yiyiyimu、murong00、massakam、mostafij-rahman、Anonymitaet、ericsyh、xuesongxs、yangl、bharanic-dev、mattisonchao、nlu90、freeznet、Technoboy-、junquero、bharanic-dev、gaoran10、Shoothzj、hangc0276、k2la、Jason918

重要进展

Broker

Broker：通过消息 ID 获取消息时打印消息元数据。 
https://github.com/apache/pulsar/pull/11092

Broker：避免向消费者暴露无意义的统计数据。 
https://github.com/apache/pulsar/pull/11005

Broker: 失败和超时的 topic 会停止 replicator 并且清理失败的 topic。 
https://github.com/apache/pulsar/pull/10847

[PIP-82] Broker：添加资源组配置监听器。
https://github.com/apache/pulsar/pull/10657

Broker: 删除 CompletableFuture 超时处理的冗余方法。 
https://github.com/apache/pulsar/pull/10970

Broker: 启用 Replicator 后，更新分区数量时不会创建 ManagedLedger。 
https://github.com/apache/pulsar/pull/10910

Broker：优化 topic 不存在时的提示信息。 
https://github.com/apache/pulsar/pull/10845

Broker：引入 metricsServletTimeoutMs 配置。 
https://github.com/apache/pulsar/pull/10886

Broker：为 geo-replicator 添加广播的 listenerName 以连接到远程集群。 
https://github.com/apache/pulsar/pull/10751

Connector

Connector：Kafka sink 连接器支持 non-primitive schema。 
https://github.com/apache/pulsar/pull/10410

Connector：使 Kinesis connector 可以从密钥加载凭据。 
https://github.com/apache/pulsar/pull/10822

Test

Test：为 TopicCountEquallyDivideBundleSplitAlgorithm添加测试。 
https://github.com/apache/pulsar/pull/11015

Test：使用 TestNG 替代 JUnit 做测试。 
https://github.com/apache/pulsar/pull/11027

Test: 增加测试 org.apache.pulsar.metadata.ZKSessionTest的稳定性。 
https://github.com/apache/pulsar/pull/10955

Test: 为 NamespaceBundleSplitAlgorithmTest 添加了一些测试。 
https://github.com/apache/pulsar/pull/10987

Test：将 JUnit 测试转换为 TestNG。 
https://github.com/apache/pulsar/pull/11020

Python

Python: 将 Python 3.9 添加到 manylinux2014 构建支持中。 
https://github.com/apache/pulsar/pull/10954

Python Functions：使用关键字参数创建 pulsar_client。 
https://github.com/apache/pulsar/pull/11080

Python Functions：使用在 function_details 中定义的订阅名称。 
https://github.com/apache/pulsar/pull/11076

C++ and Python：添加连接超时的配置。 
https://github.com/apache/pulsar/pull/11029

Security

Security：排除对 grpc-okhttp 的依赖，设置 OkHttp3 和 Okio 的版本。 
https://github.com/apache/pulsar/pull/11025

Security：将 Vertx 升级到 3.9.8 版本，以解决 CVE-2019-17640 漏洞。 
https://github.com/apache/pulsar/pull/10889

Security：将 Commons Codec 升级到 1.15 版本。 
https://github.com/apache/pulsar/pull/10864

Security: 排除并删除 FreeBuilder 依赖。
https://github.com/apache/pulsar/pull/10869

Security: 将 BouncyCastle 升级到 1.69 版本。 
https://github.com/apache/pulsar/pull/10867

Security: 将 Kubernetes Java 客户端升级到 12.0.1 版本。 
https://github.com/apache/pulsar/pull/10866

Security: 将 Caffeine 升级到 2.9.1 版本。 
https://github.com/apache/pulsar/pull/10865

Transaction

Transaction：将 managedLedger 指标添加到 Prometheus。 
https://github.com/apache/pulsar/pull/10761

Transaction：为 transaction Admin API 添加同步方法。 
https://github.com/apache/pulsar/pull/10745

其他

Admin：在 maxTopicsPerNamespace 检查中排除系统 topic。 
https://github.com/apache/pulsar/pull/10850

Authorization：删除三元操作并纠正 brokerClientAuthenticationParameter。 
https://github.com/apache/pulsar/pull/11090

Build：将 NAR 的包名称更改为 pulsar-io-kafka-connect-adaptor。 
https://github.com/apache/pulsar/pull/10976

Configuration：添加缺失的配置项。 
https://github.com/apache/pulsar/pull/11095

Metrics：使用 ZooKeeper Prometheus 指标提供器暴露 ZooKeeper 指标。 
https://github.com/apache/pulsar/pull/10803

Proxy：限制 AdminProxyHandler 中的重放缓冲区大小。 
https://github.com/apache/pulsar/pull/10944

Refactor: 为适应 Java 8 从而重构了部分方法。 
https://github.com/apache/pulsar/pull/10879

Schema Registry：即使主题上没有设置 schema， 也要让 KeyValueSchema 和 AutoConsume 组件可以工作。 
https://github.com/apache/pulsar/pull/10995

BookKeeper：将 BookKeeper 升级到 4.14.1 版本。 
https://github.com/apache/pulsar/pull/10686

Client：将 KeyValueSchema 作为公开 Schema API 接口。 
https://github.com/apache/pulsar/pull/10888

Metadata：确保 broker 之间 的元数据缓存一致。 
https://github.com/apache/pulsar/pull/10862

Functions：重新组织 Pulsar Functions 的上下文层次结构。 
https://github.com/apache/pulsar/pull/10631

Bug 修复

Broker

Broker：修复在复制的命名空间中创建分区 topic 时出现的问题。 
https://github.com/apache/pulsar/pull/10963

Broker：修复由 --precise-backlog=true参数引起的积压问题。 
https://github.com/apache/pulsar/pull/10966

Broker：修复 advertisedListener 的错误端口。 
https://github.com/apache/pulsar/pull/10961

Broker：使 PulsarClusterMetadataTeardown 的删除功能具有幂等性。 
https://github.com/apache/pulsar/pull/11042

Broker：修复错误使用的日志记录器。 
https://github.com/apache/pulsar/pull/11047

Broker：为 topic 级别的策略添加鉴权标记，以避免重定向循环。 
https://github.com/apache/pulsar/pull/11051

Broker：启用更新 ManagedLedgerConf#maximumRolloverTimeMs。 
https://github.com/apache/pulsar/pull/10746

Broker: 修复 getLastMessageId 导致直接内存泄露的问题。 
https://github.com/apache/pulsar/pull/10977

Broker: 修复 Reader 在获取最新 Message ID 时会抛出 NonRecoverableLedgerException 问题。
https://github.com/apache/pulsar/pull/10957

Broker：修复不当的 class/method/field 修饰符。 
https://github.com/apache/pulsar/pull/10837

Broker：支持自动创建系统主题 TRANSACTION_BUFFER_SNAPSHOT 。 
https://github.com/apache/pulsar/pull/10876

Broker：修复由于重复使用 entry wrapper 而导致消费者卡住的问题。 
https://github.com/apache/pulsar/pull/10824

Broker: 修复因不跳过 broker entry metadata 而引发的 parseMessageMetadata 错误。 
https://github.com/apache/pulsar/pull/10968

Broker：修复响应过滤器不能正确处理异步响应的问题。 
https://github.com/apache/pulsar/pull/11052

Broker: 修复 compaction 对系统 topic 不起作用的问题。 
https://github.com/apache/pulsar/pull/10941

Broker: 修复异常未抛出的问题。 
https://github.com/apache/pulsar/pull/10931

Build

Build：修复错误参数个数导致的编译错误。 
https://github.com/apache/pulsar/pull/11104

Build：支持在未设置 CROWDIN_DOCUSAURUS_API_KEY 的情况下构建 Pulsar。 
https://github.com/apache/pulsar/pull/10804

Build: 将 io.airlift.aircompressor 更新到 版本 0.19 。 
https://github.com/apache/pulsar/pull/10983

Build: 修复关于 Pulsar Functions 的默认 main 配置文件激活的问题。 
https://github.com/apache/pulsar/pull/10887

Client

Client：取消订阅 topic 时，释放 multiTopicsConsumerImpl。 
https://github.com/apache/pulsar/pull/10997

Client：删除 Pulsar Functions 实例 JAR 中不需要的依赖项，并将 SchemaInfo 作为接口。 
https://github.com/apache/pulsar/pull/10878

Java Client：在调用 redeliverUnacknowledgedMessages 之前释放  UnAckedMessageTracker 的写锁。 
https://github.com/apache/pulsar/pull/10768

Client: 修复 ConsumerImpl#internalGetLastMessageIdAsync无返回的问题。 
https://github.com/apache/pulsar/pull/10974

Client: 修复非持久化 topic 在 discovery 时获得分区元数据的问题。 
https://github.com/apache/pulsar/pull/10806

C++

C++：修复 Windows 32 位版本 C++ 源代码编译不成功的问题。 
https://github.com/apache/pulsar/pull/11082

C++：修复 KeySharedMeta 未设置为订阅命令的问题。 
https://github.com/apache/pulsar/pull/11088

C++：修复未初始化变量导致的未知行为。
https://github.com/apache/pulsar/pull/10892

C++: 修复关于静态库文件中 windows 构建的问题。 
https://github.com/apache/pulsar/pull/10956

Test

Test：修复 testEnableAndDisableTopicDelayedDeliveryFlaky Test。 
https://github.com/apache/pulsar/pull/11009

Test：修复 ManagedLedgerTest Flaky Test。 
https://github.com/apache/pulsar/pull/11016

Test：修复 testDuplicateConcurrentSubscribeCommandFlaky Test。 
https://github.com/apache/pulsar/pull/11089

Test：修复 PrimitiveSchemaTest Flaky Test。 
https://github.com/apache/pulsar/pull/11038

Test：修复 AdminApiTest Flaky Test。 
https://github.com/apache/pulsar/pull/11039

Test：修复用 TestNG 替换 JUnit 时 assertEquals 方法参数顺序改变的问题。 
https://github.com/apache/pulsar/pull/11035

Test：修复 AdminApiOffloadTest Flaky Test。 
https://github.com/apache/pulsar/pull/11028

Test：修复 ResourceGroupConfigListenerTest Flaky Test. 
https://github.com/apache/pulsar/pull/11048

Test：修复 AdminApiTest Flaky Test。 
https://github.com/apache/pulsar/pull/11059

Test：修复 testTransactionMetaStoreAssignAndFailoverFlaky Test。 
https://github.com/apache/pulsar/pull/11073

Topic Policy

Topic Policy：修复尚未触发主题压缩时系统 topic 上的潜在数据丢失。 
https://github.com/apache/pulsar/pull/11003

Topic Policy：修复 topic 级别保留策略不起作用的问题。 
https://github.com/apache/pulsar/pull/11021

Topic Policy：修复 SystemTopicBasedTopicPoliciesService 的竞态条件。 
https://github.com/apache/pulsar/pull/11097

Functions

Functions：在 Java Function 实例中引入 Common-IO。 
https://github.com/apache/pulsar/pull/10939

Functions：修复 SpotBugs 引起的构建失败。 
https://github.com/apache/pulsar/pull/10792

Transaction

Transaction：修复加载 namespace 包时 broker 初始化事务相关 topic 的问题。 
https://github.com/apache/pulsar/pull/11022

Transaction：修复 transaction Admin API 的路径传递参数问题。 
https://github.com/apache/pulsar/pull/10748

Transactions: 防止在没有成功执行 startAsync() 的情况下调用 closeAsync() 时发生 空指针异常。 
https://github.com/apache/pulsar/pull/10948

其他

Python：修复不使用 AvroSchema 时导入 Pulsar 2.8.0 Python 客户端的错误。 
https://github.com/apache/pulsar/pull/11034

Pulsar SQL：修复对 primary schema 的表运行 select count(*) 时发生的 Pulsar SQL 问题。 
https://github.com/apache/pulsar/pull/10840

Schema：修复 AutoConsumeSchema 使用空 schema 版本解码消息时发生的 NEP。 
https://github.com/apache/pulsar/pull/10811

Schema：修复死信 producer 不处理 AUTO_CONSUME_SCHEMA 的问题。 
https://github.com/apache/pulsar/pull/9970

Kinesis：修复 Kinesis sink 连接器不确认消息的问题。 
https://github.com/apache/pulsar/pull/10769

Kinesis：修复找不到 Kinesis sink Backoff 类的问题。 
https://github.com/apache/pulsar/pull/10744

Common: 更新 TopicName#getPartitionIndex 的实现逻辑。 
https://github.com/apache/pulsar/pull/10902

BookKeeper：遍历 batchDeletedIndexes 时添加读锁。 
https://github.com/apache/pulsar/pull/10763

Proxy: 修复字符串在循环中的连接问题。 
https://github.com/apache/pulsar/pull/10986

Metadata: 删除重复的 awaitility 依赖。 
https://github.com/apache/pulsar/pull/10979

CLI: 修复对 list.remove 方法的错误使用。 
https://github.com/apache/pulsar/pull/10935

活动与干货

活动

•TGIP：关于 Pulsar 及其生态系统的每周直播^[1]•Intro to Apache Pulsar 101：关于 Pulsar 的每月直播，包括概念、架构和设计^[2]•Highlights from The Pulsar Virtual Summit North America 2021^[3]•视频回顾 | Pulsar Summit 2021 北美峰会 - 主题演讲•ApacheCon Asia 2021: Apache Pulsar 技术议题一览•QCon 北京｜Apache Pulsar：云原生时代的消息服务

案例

•Apache Pulsar 助力金山云日志服务，日处理 200TB 数据•最佳实践｜放弃 Ceph，Salesforce 使用 Apache BookKeeper 在云中实现最强存储•最佳实践｜Apache Pulsar 在华为云物联网之旅

博文

•博文推荐 | 基于 Pulsar 事务实现 Exactly-Once 语义•Apache Pulsar：云原生时代的消息服务•直播回顾 | TGIP-CN 034：RoP 的前世今生(附视频与PPT)•Four Reasons Why Apache Pulsar is Essential to the Modern Data Stack^[4]•Apache Pulsar Demonstrates Best-in-Class Cloud-Native Price-Performance^[5]•Building Connectors On Pulsar Made Simple^[6]•What’s New in Apache Pulsar 2.6.4^[7]•Pulsar User Survey 2021 Highlights^[8]•2021 Apache Pulsar User Report Announcement^[9]•Introducing StreamNative Platform^[10]•StreamNative Platform, An Enterprise Messaging and Streaming Platform Powered by Apache Pulsar^[11]•Apache Pulsar Launches 2.8: Unified Messaging and Streaming With Transactions^[12]•Why developers should use Apache Pulsar -- Patrick McFadin^[13]

生态

•庆祝：Apache Pulsar 社区新里程碑——全球贡献者超 400 位！•喜讯 | KoP 开发工程师徐昀泽入选 Apache Pulsar Committer

————————

以上就是 2021 年 6 月份的脉动之旅，在此感谢参与周报与博客翻译整理的社区志愿者们：资飞、王嘉凌、刘梓霖、王伦辉、shaohan.niu，本月报内容也是基于上述志愿者的贡献汇总而成，再次感谢！如果你也想参与我们的社区志愿工作，欢迎加入！

Apache Pulsar 社区鼓励大家积极参与开源社区，无论是文档、代码、翻译，还是技术博客，都欢迎大家积极参与，早日成为 Pulsar contributor，一起加油鸭。

相关阅读

•  Pulsar 社区周报｜2021-06-21～2021-06-27

•  Pulsar 社区周报｜2021-06-14～2021-06-20

•  Pulsar 社区周报｜2021-06-07～2021-06-13

引用链接

[1] TGIP：关于 Pulsar 及其生态系统的每周直播: https://streamnative.io/resource#tgip
[2] Intro to Apache Pulsar 101：关于 Pulsar 的每月直播，包括概念、架构和设计: https://streamnative.io/en/resource#intro-to-apache-pulsar-101
[3] Highlights from The Pulsar Virtual Summit North America 2021: https://streamnative.io/en/blog/community/2021-06-22-highlights-from-the-pulsar-virtual-summit-north-america-2021
[4] Four Reasons Why Apache Pulsar is Essential to the Modern Data Stack: https://datastax.medium.com/four-reasons-why-apache-pulsar-is-essential-to-the-modern-data-stack-b90a8bddcb9
[5] Apache Pulsar Demonstrates Best-in-Class Cloud-Native Price-Performance: https://gigaom.com/2021/06/11/apache-pulsar-demonstrates-best-in-class-cloud-native-price-performance/
[6] Building Connectors On Pulsar Made Simple: https://streamnative.io/en/blog/tech/2021-06-01-building-connectors-on-pulsar-made-simple
[7] What’s New in Apache Pulsar 2.6.4: https://streamnative.io/en/blog/release/2021-06-08-pulsar-264
[8] Pulsar User Survey 2021 Highlights: https://streamnative.io/en/blog/community/2021-06-11-pulsar-user-survey-2021-highlights
[9] 2021 Apache Pulsar User Report Announcement: https://streamnative.io/en/blog/community/2021-06-10-2021-apache-pulsar-user-report-announcement
[10] Introducing StreamNative Platform: https://streamnative.io/en/blog/release/2021-06-16-streamnative-platform
[11] StreamNative Platform, An Enterprise Messaging and Streaming Platform Powered by Apache Pulsar: https://www.pressrelease.cc/2021/06/17/streamnative-platform-an-enterprise-messaging-and-streaming-platform-powered-by-apache-pulsar/
[12] Apache Pulsar Launches 2.8: Unified Messaging and Streaming With Transactions: https://streamnative.io/en/blog/release/2021-06-15-apache-pulsar-launches-2-8-unified-messaging-and-streaming-with-transactions
[13] Why developers should use Apache Pulsar -- Patrick McFadin: https://www.infoworld.com/article/3619272/why-developers-should-use-apache-pulsar.html#tk.rss_all

点击“阅读原文”，获取 Apache Pulsar 硬核干货资料！

‍