单机版kafka集群有什么作用

练习上手用。

搭建zookeeper集群首先下载zookeeper解压 apache zookeeper

tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz将zookeeper复制三份,分别命名为zookeeper-1,zookeeper-2,zookeeper-3

将zookeeper-1中的zoo.example.cfg文件复制一份改名为: zoo.cfg

修改config/zoo.cfg文件修改端口: clientPort=2181

修改数据目录: dataDir=/ashura/zookeeper-1/datalog

增加以下配置:

server.1=localhost.:2887:3887 server.2=localhost.:2888:3888 server.3=localhost.:2889:3889 admin.serverPort=8000

完成的配置文件如下:

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/ashura/zookeeper-1/datalog

# the port at which the clients will connect

clientPort=2181

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

server.1=localhost.:2887:3887

server.2=localhost.:2888:3888

server.3=localhost.:2889:3889将这份zoo.cfg分别复制到zookeeper-2,zookeeper-3的config目录下.修改zookeeper2的zoo.cfg中clientPort=2183,dataDir=/ashura/zookeeper-2/datalog

修改zookeeper3的zoo.cfg中clientPort=2184,dataDir=/ashura/zookeeper-3/datalog

创建刚才在配置文件中写的目录

mkdir /ashura/zookeeper-1/datalog

mkdir /ashura/zookeeper-2/datalog

mkdir /ashura/zookeeper-3/datalog分别{-- 在datalog目录下 --}执行以下命令,写入myid。

echo "1" > /ashura/zookeeper-1/datalog/myid

echo "2" > /ashura/zookeeper-2/datalog/myid

echo "3" > /ashura/zookeeper-3/datalog/myid最后分别启动zookeeper集群

/ashura/zookeeper-1/bin/zkServer.sh start

/ashura/zookeeper-2/bin/zkServer.sh start

/ashura/zookeeper-3/bin/zkServer.sh start

使用如下命令判断是否启动成功

/ashura/zookeeper-1/bin/zkServer.sh status

/ashura/zookeeper-2/bin/zkServer.sh status

/ashura/zookeeper-3/bin/zkServer.sh status

搭建Kafka集群

下载

开始安装解压

tar -zxvf kafka_2.11-2.2.1.tgz将config/server.properties复制三份，分别命名为server1.properties,server2.properties,server3.properties。

修改server1.propertiesbroker.id=1

listeners=PLAINTEXT://:9092

advertised.listeners=PLAINTEXT://10.1.14.159:9092(其中10.1.14.159是我本机的ip)

log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka1-logs

zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

同理,修改server2.propertiesbroker.id=2

listeners=PLAINTEXT://:9093

advertised.listeners=PLAINTEXT://10.1.14.159:9093(其中10.1.14.159是我本机的ip)

log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka2-logs

zookeeper.connect=localhost:2181,localhost:2182,localhost:2183同理,修改server3.propertiesbroker.id=3

listeners=PLAINTEXT://:9094

advertised.listeners=PLAINTEXT://10.1.14.159:9094(其中10.1.14.159是我本机的ip)

log.dirs=/ashura/kafka_2.11-2.2.1/logs/kafka3-logs

zookeeper.connect=localhost:2181,localhost:2182,localhost:2183

然后执行以下命令

nohup /ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh /ashura/kafka_2.11-2.2.1/config/server3.properties > /ashura/kafka_2.11-2.2.1/logs/kafka3-logs/startup.log 2>&1 &

nohup /ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh /ashura/kafka_2.11-2.2.1/config/server2.properties > /ashura/kafka_2.11-2.2.1/logs/kafka2-logs/startup.log 2>&1 &

nohup /ashura/kafka_2.11-2.2.1/bin/kafka-server-start.sh /ashura/kafka_2.11-2.2.1/config/server1.properties > /ashura/kafka_2.11-2.2.1/logs/kafka1-logs/startup.log 2>&1 &通过startup.log,或者同级目录下的server.log查看是否有报错即可。

检测创建主题:./kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --create --topic fxb_test1 --replication-factor 3 --partitions 3

启动消费者: ./kafka-console-producer.sh --broker-list 10.1.14.159:9092 --topic fxb_test1

新开窗口个,启动生产者: kafka-console-producer.sh --bootstrap-server 127.0.0.1 --create --topic fxb_test1 在生产者窗口中输入消息，查看消费者的窗口，是否有消息产生。

参考文档

使用java client进行测试引入依赖

org.apache.kafka

kafka-clients

2.3.0

创建生产者

package com.fxb.learn.kafka.producer;

import org.apache.kafka.clients.producer.KafkaProducer;

import org.apache.kafka.clients.producer.Producer;

import org.apache.kafka.clients.producer.ProducerConfig;

import org.apache.kafka.clients.producer.ProducerRecord;

import java.util.Properties;

/***

* 生产者

*/

public class ProducerDemo {

public static void main(String[] args) {

Properties props = new Properties();

props.put("bootstrap.servers", "10.127.138.75:9092");

props.put("acks", "all");

props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");

props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

// props.put(ProducerConfig.Re)

Producer producer = new KafkaProducer<>(props);

for (int i = 0; i < 10; i++) {

producer.send(new ProducerRecord("fxb_test1", Integer.toString(i), Integer.toString(i)));

System.out.println("has sent msg [" + i + "]");

}

producer.close();

}

}创建消费者

package com.fxb.learn.kafka.consumer;

import org.apache.kafka.clients.consumer.ConsumerRecord;

import org.apache.kafka.clients.consumer.ConsumerRecords;

import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.time.Duration;

import java.util.Arrays;

import java.util.Properties;

/***

* 消费者

*/

public class ConsumerDemo {

public static void main(String[] args) {

Properties props = new Properties();

props.setProperty("bootstrap.servers", "10.127.138.75:9092");

props.setProperty("group.id", "test");

props.setProperty("enable.auto.commit", "true");

props.setProperty("auto.commit.interval.ms", "1000");

props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer consumer = new KafkaConsumer<>(props);

consumer.subscribe(Arrays.asList("fxb_test1", "bar"));

while (true) {

ConsumerRecords records = consumer.poll(Duration.ofMillis(100));

for (ConsumerRecord record : records)

System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());

}

}

}