Kafka-Listener配置参数、配置公网访问以及可能出现的问题

文章目录

核心参数
- listeners
- advertised.listeners
- listener.security.protocol.map
- inter.broker.listener.name
配置外网访问
- 配置实例
- 可能会出现的问题
- - listener_name不能相同
  - inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are PRIVATE,PUBLIC
  - advertised.listeners listener names must be equal to or a subset of the ones defined in listeners

核心参数

listeners

Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol, listener.security.protocol.map must also be set. Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface. Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093

简要说明
配置需要监听的URI

参数格式

使用逗号分隔的URI列表
协议名称://主机名:端口、listener名称://主机名:端口
不同主机名可以绑定相同端口

附加说明

如果列表中存在使用相同协议的listener，必须要在listener.security.protocol.map中定义不同的名称
可以使用0.0.0.0的ip绑定所有网卡

advertised.listeners

Listeners to publish to ZooKeeper for clients to use, if different than the listeners config property. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used. Unlike listeners it is not valid to advertise the 0.0.0.0 meta-address.
简要说明
该参数是将listener发布到Zk中，供client使用

参数格式

参数值必须为listeners中定义的某个listener 或者对应name的listener在listeners中的定义使用了0.0.0.0作为host，在advertised.listeners中可以指定某个网卡ip
不能使用host为0.0.0.0的listener

listener.security.protocol.map

Map between listener names and security protocols. This must be defined for the same security protocol to be usable in more than one port or IP. For example, internal and external traffic can be separated even if SSL is required for both. Concretely, the user could define listeners with names INTERNAL and EXTERNAL and this property as: INTERNAL:SSL,EXTERNAL:SSL. As shown, key and value are separated by a colon and map entries are separated by commas. Each listener name should only appear once in the map. Different security (SSL and SASL) settings can be configured for each listener by adding a normalised prefix (the listener name is lowercased) to the config name. For example, to set a different keystore for the INTERNAL listener, a config with name listener.name.internal.ssl.keystore.location would be set. If the config for the listener name is not set, the config will fallback to the generic config (i.e. ssl.keystore.location)

简单说明
该参数用于listener name与安全协议之间进行映射。

参数格式
键值是通过冒号进行分隔的，键值对之间通过逗号进行分隔。协议的合法取值为：PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL

附加说明

安全参数的配置通过添加前缀进行设置，比如配置LISTENER_A的ssl.keystore.location的参数：listener.name.listener_a.ssl.keystore.location，listener name需要转成小写。

inter.broker.listener.name

Name of listener used for communication between brokers. If this is unset, the listener name is defined by security.inter.broker.protocol. It is an error to set this and security.inter.broker.protocol properties at the same time

简单说明
该参数是用来设置broker之间进行通信时采用的listener名称
参数格式
listener name
附加说明
如果该参数没有设置，则采用security.inter.broker.protocol配置。同时设置inter.broker.listener.name和security.inter.broker.protocol会出现错误。listener必须要在advertised.listeners中进行配置，否则会出现inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are A,B的错误。

配置外网访问

配置实例

listeners=PRIVATE://0.0.0.0:9092,PUBLIC://0.0.0.0:9093
advertised.listeners=PRIVATE://192.168.16.4:9092,PUBLIC://180.77.248.246:9093
listener.security.protocol.map=PRIVATE:PLAINTEXT,PUBLIC:PLAINTEXT
inter.broker.listener.name=PRIVATE

可能会出现的问题

listener_name不能相同

这个是因为我使用了PLAINTEXT作为多个listener的名称，但是当使用协议名称作为listener名称时，只能使用一次，因为本身listener名称是不能重复。一般情况下，我们还是自己在listener.security.protocol.map中进行listener name和协议的映射。

inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are PRIVATE,PUBLIC

这是因为我使用了自定义的listener name定义的listener，一般来说默认inter.broker.listener.name为PLAINTEXT，如果我们在listener列表中没有定义名称为PLAINTEXT的listener，就会抛出该异常。

advertised.listeners listener names must be equal to or a subset of the ones defined in listeners

该问题是我误解了listener host的意义导致的，我一开始的理解是，该host是用于过滤客户机的ip地址，只有用于定义listener的客户机ip才能进行连接，有点白名单的意思，listeners是内网ip白名单，advertised.listeners是外网ip白名单，这个理解当然是错误的。我们在定义时，必须遵照规则：

参数值必须为listeners中定义的某个listener 或者对应name的listener在listeners中的定义使用了0.0.0.0作为host，在advertised.listeners中可以指定某个网卡ip

参考来源：
1、https://www.codercto.com/a/68756.html
2、http://kafka.apache.org/documentation/#configuration