Windows PC上创建大数据职业技能竞赛实验环境之六--Flume、Kafka和Flink编程

1 Flume

参看日志采集工具Flume的安装与使用方法_厦大数据库实验室博客 (xmu.edu.cn)。

查看Flume安装

root@client1:~# flume-ng version
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523

Avro测试

创建agent配置文件。

root@client1:~# cd hadoop/apache-flume*/conf
root@client1:~/hadoop/apache-flume-1.7.0-bin/conf# vi avro.conf

  a1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.channels = c1a1.sources.r1.bind = 0.0.0.0a1.sources.r1.port = 4141
#注意这个端口名，在后面的教程中会用得到# Describe the sinka1.sinks.k1.type = logger# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

启动flume agent a1

root@client1:~/hadoop/apache-flume-1.7.0-bin/conf# cd
root@client1:~# flume-ng agent -c . -f hadoop/apache-flume-1.7.0-bin/conf/avro.conf -n a1 -Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via (/root/hadoop/hadoop-2.7.7/bin/hadoop) for HDFS access
Info: Including Hive libraries found via (/root/hadoop/apache-hive-2.3.4-bin) for Hive access
+ exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/root:/root/hadoop/apache-flume-1.7.0-bin/lib/*:/root/hadoop/hadoop-2.7.7/etc/hadoop:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/common/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/*:/root/hadoop/hadoop-2.7.7/contrib/capacity-scheduler/*.jar:/root/hadoop/apache-hive-2.3.4-bin/lib/*' -Djava.library.path=:/root/hadoop/hadoop-2.7.7/lib/native org.apache.flume.node.Application -f hadoop/apache-flume-1.7.0-bin/conf/avro.conf -n a1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/apache-flume-1.7.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
22/04/15 09:07:50 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
22/04/15 09:07:50 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:hadoop/apache-flume-1.7.0-bin/conf/avro.conf
22/04/15 09:07:50 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
22/04/15 09:07:50 INFO conf.FlumeConfiguration: Processing:k1
22/04/15 09:07:50 INFO conf.FlumeConfiguration: Processing:k1
22/04/15 09:07:50 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
22/04/15 09:07:50 INFO node.AbstractConfigurationProvider: Creating channels
22/04/15 09:07:50 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type memory
22/04/15 09:07:50 INFO node.AbstractConfigurationProvider: Created channel c1
22/04/15 09:07:50 INFO source.DefaultSourceFactory: Creating instance of source r1, type avro
22/04/15 09:07:50 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: logger
22/04/15 09:07:50 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]
22/04/15 09:07:50 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: 0.0.0.0, port: 4141 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@441ddd7c counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
22/04/15 09:07:50 INFO node.Application: Starting Channel c1
22/04/15 09:07:50 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
22/04/15 09:07:50 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
22/04/15 09:07:50 INFO node.Application: Starting Sink k1
22/04/15 09:07:50 INFO node.Application: Starting Source r1
22/04/15 09:07:50 INFO source.AvroSource: Starting Avro source r1: { bindAddress: 0.0.0.0, port: 4141 }...
22/04/15 09:07:50 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
22/04/15 09:07:50 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
22/04/15 09:07:50 INFO source.AvroSource: Avro source r1 started.

打开另外一个终端，在/usr/local/flume下写入一个文件log.00。

root@client1:~# echo "hello world" > log.00

root@client1:~# flume-ng avro-client --conf conf -H localhost -p 4141 -F ./log.00
Info: Including Hadoop libraries found via (/root/hadoop/hadoop-2.7.7/bin/hadoop) for HDFS access
Info: Including Hive libraries found via (/root/hadoop/apache-hive-2.3.4-bin) for Hive access
+ exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xmx20m -cp 'conf:/root/hadoop/apache-flume-1.7.0-bin/lib/*:/root/hadoop/hadoop-2.7.7/etc/hadoop:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/common/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/*:/root/hadoop/hadoop-2.7.7/contrib/capacity-scheduler/*.jar:/root/hadoop/apache-hive-2.3.4-bin/lib/*' -Djava.library.path=:/root/hadoop/hadoop-2.7.7/lib/native org.apache.flume.client.avro.AvroCLIClient -H localhost -p 4141 -F ./log.00
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/apache-flume-1.7.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
22/04/15 09:14:25 WARN api.NettyAvroRpcClient: Using default maxIOWorkers
root@client1:~#

第一个终端（agent窗口）下的显示，也就是在日志控制台，就会把log.00文件的内容打印出来。

22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 => /127.0.0.1:4141] OPEN
22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 => /127.0.0.1:4141] BOUND: /127.0.0.1:4141
22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 => /127.0.0.1:4141] CONNECTED: /127.0.0.1:48154
22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 :> /127.0.0.1:4141] DISCONNECTED
22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 :> /127.0.0.1:4141] UNBOUND
22/04/15 09:14:25 INFO ipc.NettyServer: [id: 0x26eaff20, /127.0.0.1:48154 :> /127.0.0.1:4141] CLOSED
22/04/15 09:14:25 INFO ipc.NettyServer: Connection to /127.0.0.1:48154 disconnected.
22/04/15 09:14:28 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64                hello world }

netcatsource测试

创建agent配置文件。

root@client1:~# cd hadoop/apache-flume*/conf
root@client1:~/hadoop/apache-flume-1.7.0-bin/conf# vi netsource.conf

启动flume agent (即打开日志控制台)。

root@client1:~/hadoop/apache-flume-1.7.0-bin/conf# cd
root@client1:~# flume-ng agent -c . -f hadoop/apache-flume-1.7.0-bin/conf/netsource.conf -n a1 -Dflume.root.logger=INFO,
console
Info: Including Hadoop libraries found via (/root/hadoop/hadoop-2.7.7/bin/hadoop) for HDFS access
Info: Including Hive libraries found via (/root/hadoop/apache-hive-2.3.4-bin) for Hive access
+ exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/root:/root/hadoop/apache-flume-1.7.0-bin/lib/*:/root/hadoop/hadoop-2.7.7/etc/hadoop:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/common/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/hdfs/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/yarn/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/lib/*:/root/hadoop/hadoop-2.7.7/share/hadoop/mapreduce/*:/root/hadoop/hadoop-2.7.7/contrib/capacity-scheduler/*.jar:/root/hadoop/apache-hive-2.3.4-bin/lib/*' -Djava.library.path=:/root/hadoop/hadoop-2.7.7/lib/native org.apache.flume.node.Application -f hadoop/apache-flume-1.7.0-bin/conf/netsource.conf -n a1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/apache-flume-1.7.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/hadoop/apache-hive-2.3.4-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
22/04/15 09:26:23 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
22/04/15 09:26:23 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:hadoop/apache-flume-1.7.0-bin/conf/netsource.conf
22/04/15 09:26:23 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
22/04/15 09:26:23 INFO conf.FlumeConfiguration: Processing:k1
22/04/15 09:26:23 INFO conf.FlumeConfiguration: Processing:k1
22/04/15 09:26:23 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
22/04/15 09:26:23 INFO node.AbstractConfigurationProvider: Creating channels
22/04/15 09:26:23 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type memory
22/04/15 09:26:23 INFO node.AbstractConfigurationProvider: Created channel c1
22/04/15 09:26:23 INFO source.DefaultSourceFactory: Creating instance of source r1, type netcat
22/04/15 09:26:23 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: logger
22/04/15 09:26:23 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]
22/04/15 09:26:23 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:r1,state:IDLE} }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2631fbbf counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
22/04/15 09:26:23 INFO node.Application: Starting Channel c1
22/04/15 09:26:23 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
22/04/15 09:26:23 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
22/04/15 09:26:23 INFO node.Application: Starting Sink k1
22/04/15 09:26:23 INFO node.Application: Starting Source r1
22/04/15 09:26:23 INFO source.NetcatSource: Source starting
22/04/15 09:26:23 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

再打开一个终端，输入命令:telnet localhost 44444。

root@client1:~# telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

如果终端提示没有安装telnet，使用apt install telnet安装。

然后我们可以在终端下输入任何字符，第一个终端的日志控制台也会有相应的显示，如我们输入”hello,world”。

root@client1:~# telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Hello world
OK

第一个终端的日志控制台显示。

22/04/15 09:32:29 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 0D             Hello world. }

2 启动Zookeeper

我们编辑一个shell脚本，方便一次性启动各容器节点的zookeeper服务，start-zookeeper.sh。

#!/bin/bashhosts=("master" "node02" "node03")
user="root"i=0
for host in ${hosts[*]}
doi=$(expr $i + 1)echo "$host:"if [ $i -eq "1" ]; then/root/hadoop/apache-zookeeper-3.6.3-bin/bin/zkServer.sh startelsessh $user@$host > /dev/null 2>&1 << eeooffsed -i "s/1/$i/g" /root/hadoop/apache-zookeeper-3.6.3-bin/zkData/myid
/root/hadoop/apache-zookeeper-3.6.3-bin/bin/zkServer.sh start
exit
eeoofffi
done

3 Kafka

参见Kafka的安装和简单实例测试_厦大数据库实验室博客 (xmu.edu.cn)。

在一个终端运行。

root@client1:~# zookeeper-server-start.sh hadoop/kafka_2.11-2.0.0/config/zookeeper.properties
[2022-04-15 13:40:35,909] INFO Reading configuration from: hadoop/kafka_2.11-2.0.0/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-15 13:40:35,912] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-15 13:40:35,912] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-15 13:40:35,912] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2022-04-15 13:40:35,912] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2022-04-15 13:40:35,926] INFO Reading configuration from: hadoop/kafka_2.11-2.0.0/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2022-04-15 13:40:35,926] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2022-04-15 13:40:35,935] INFO Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,935] INFO Server environment:host.name=client1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,935] INFO Server environment:java.version=1.8.0_312 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,935] INFO Server environment:java.vendor=Private Build (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,935] INFO Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:java.class.path=.:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib:/usr/lib/jvm/java-8-openjdk-amd64/lib:/usr/lib/jvm/java-8-openjdk-amd64/lib/dt.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/activation-1.1.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/argparse4j-0.7.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/audience-annotations-0.5.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/commons-lang3-3.5.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-api-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-basic-auth-extension-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-file-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-json-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-runtime-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-transforms-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/guava-20.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-api-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-locator-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-utils-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-annotations-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-core-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-databind-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-jaxrs-base-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-jaxrs-json-provider-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-module-jaxb-annotations-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javassist-3.22.0-CR2.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.annotation-api-1.2.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.inject-1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.inject-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.servlet-api-3.1.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.ws.rs-api-2.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jaxb-api-2.3.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-client-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-common-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-container-servlet-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-container-servlet-core-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-hk2-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-media-jaxb-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-server-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-client-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-continuation-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-http-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-io-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-security-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-server-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-servlet-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-servlets-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-util-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jopt-simple-5.0.4.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-clients-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-log4j-appender-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-examples-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-scala_2.11-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-test-utils-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-tools-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka_2.11-2.0.0-sources.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka_2.11-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/log4j-1.2.17.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/lz4-java-1.4.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/maven-artifact-3.5.3.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/metrics-core-2.2.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/plexus-utils-3.1.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/reflections-0.9.11.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/rocksdbjni-5.7.3.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-library-2.11.12.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-logging_2.11-3.9.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-reflect-2.11.12.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/slf4j-api-1.7.25.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/slf4j-log4j12-1.7.25.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/snappy-java-1.1.7.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/validation-api-1.1.0.Final.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/zkclient-0.10.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/zookeeper-3.4.13.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:os.version=5.10.102.1-microsoft-standard-WSL2 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,936] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,937] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,937] INFO Server environment:user.dir=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,945] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,946] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,946] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2022-04-15 13:40:35,953] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2022-04-15 13:40:35,957] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

另终端运行。

root@client1:~# kafka-server-start.sh hadoop/kafka_2.11-2.0.0/config/server.properties
[2022-04-15 13:47:13,172] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-04-15 13:47:13,372] INFO starting (kafka.server.KafkaServer)
[2022-04-15 13:47:13,372] INFO Connecting to zookeeper on master:2181,node02:2181,node03:2181 (kafka.server.KafkaServer)
[2022-04-15 13:47:13,389] INFO [ZooKeeperClient] Initializing a new session to master:2181,node02:2181,node03:2181. (kafka.zookeeper.ZooKeeperClient)
[2022-04-15 13:47:13,404] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,404] INFO Client environment:host.name=client1 (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,404] INFO Client environment:java.version=1.8.0_312 (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,404] INFO Client environment:java.vendor=Private Build (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,404] INFO Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,404] INFO Client environment:java.class.path=.:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib:/usr/lib/jvm/java-8-openjdk-amd64/lib:/usr/lib/jvm/java-8-openjdk-amd64/lib/dt.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/activation-1.1.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/aopalliance-repackaged-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/argparse4j-0.7.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/audience-annotations-0.5.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/commons-lang3-3.5.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-api-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-basic-auth-extension-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-file-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-json-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-runtime-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/connect-transforms-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/guava-20.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-api-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-locator-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/hk2-utils-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-annotations-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-core-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-databind-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-jaxrs-base-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-jaxrs-json-provider-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jackson-module-jaxb-annotations-2.9.6.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javassist-3.22.0-CR2.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.annotation-api-1.2.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.inject-1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.inject-2.5.0-b42.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.servlet-api-3.1.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/javax.ws.rs-api-2.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jaxb-api-2.3.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-client-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-common-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-container-servlet-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-container-servlet-core-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-hk2-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-media-jaxb-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jersey-server-2.27.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-client-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-continuation-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-http-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-io-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-security-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-server-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-servlet-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-servlets-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jetty-util-9.4.11.v20180605.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/jopt-simple-5.0.4.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-clients-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-log4j-appender-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-examples-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-scala_2.11-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-streams-test-utils-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka-tools-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka_2.11-2.0.0-sources.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/kafka_2.11-2.0.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/log4j-1.2.17.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/lz4-java-1.4.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/maven-artifact-3.5.3.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/metrics-core-2.2.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/plexus-utils-3.1.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/reflections-0.9.11.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/rocksdbjni-5.7.3.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-library-2.11.12.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-logging_2.11-3.9.0.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/scala-reflect-2.11.12.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/slf4j-api-1.7.25.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/slf4j-log4j12-1.7.25.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/snappy-java-1.1.7.1.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/validation-api-1.1.0.Final.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/zkclient-0.10.jar:/root/hadoop/kafka_2.11-2.0.0/bin/../libs/zookeeper-3.4.13.jar (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,405] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:os.version=5.10.102.1-microsoft-standard-WSL2 (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,406] INFO Client environment:user.dir=/root (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,409] INFO Initiating client connection, connectString=master:2181,node02:2181,node03:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@38d8f54a (org.apache.zookeeper.ZooKeeper)
[2022-04-15 13:47:13,418] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2022-04-15 13:47:13,419] INFO Opening socket connection to server node03/172.17.0.4:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-04-15 13:47:13,424] INFO Socket connection established to node03/172.17.0.4:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-04-15 13:47:13,447] INFO Session establishment complete on server node03/172.17.0.4:2181, sessionid = 0x3000102e4b00001, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2022-04-15 13:47:13,449] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
[2022-04-15 13:47:13,678] INFO Cluster ID = oXKnJRY-Qq6PLa2T3MsNJw (kafka.server.KafkaServer)
[2022-04-15 13:47:13,681] WARN No meta.properties file under dir /root/hadoop/kafka_2.11-2.0.0/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2022-04-15 13:47:13,719] INFO KafkaConfig values:advertised.host.name = nulladvertised.listeners = nulladvertised.port = nullalter.config.policy.class.name = nullalter.log.dirs.replication.quota.window.num = 11alter.log.dirs.replication.quota.window.size.seconds = 1authorizer.class.name =auto.create.topics.enable = trueauto.leader.rebalance.enable = truebackground.threads = 10broker.id = 0broker.id.generation.enable = truebroker.rack = nullclient.quota.callback.class = nullcompression.type = producerconnections.max.idle.ms = 600000controlled.shutdown.enable = truecontrolled.shutdown.max.retries = 3controlled.shutdown.retry.backoff.ms = 5000controller.socket.timeout.ms = 30000create.topic.policy.class.name = nulldefault.replication.factor = 1delegation.token.expiry.check.interval.ms = 3600000delegation.token.expiry.time.ms = 86400000delegation.token.master.key = nulldelegation.token.max.lifetime.ms = 604800000delete.records.purgatory.purge.interval.requests = 1delete.topic.enable = truefetch.purgatory.purge.interval.requests = 1000group.initial.rebalance.delay.ms = 0group.max.session.timeout.ms = 300000group.min.session.timeout.ms = 6000host.name =inter.broker.listener.name = nullinter.broker.protocol.version = 2.0-IV1leader.imbalance.check.interval.seconds = 300leader.imbalance.per.broker.percentage = 10listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSLlisteners = nulllog.cleaner.backoff.ms = 15000log.cleaner.dedupe.buffer.size = 134217728log.cleaner.delete.retention.ms = 86400000log.cleaner.enable = truelog.cleaner.io.buffer.load.factor = 0.9log.cleaner.io.buffer.size = 524288log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308log.cleaner.min.cleanable.ratio = 0.5log.cleaner.min.compaction.lag.ms = 0log.cleaner.threads = 1log.cleanup.policy = [delete]log.dir = /tmp/kafka-logslog.dirs = /root/hadoop/kafka_2.11-2.0.0/kafka-logslog.flush.interval.messages = 9223372036854775807log.flush.interval.ms = nulllog.flush.offset.checkpoint.interval.ms = 60000log.flush.scheduler.interval.ms = 9223372036854775807log.flush.start.offset.checkpoint.interval.ms = 60000log.index.interval.bytes = 4096log.index.size.max.bytes = 10485760log.message.downconversion.enable = truelog.message.format.version = 2.0-IV1log.message.timestamp.difference.max.ms = 9223372036854775807log.message.timestamp.type = CreateTimelog.preallocate = falselog.retention.bytes = -1log.retention.check.interval.ms = 300000log.retention.hours = 168log.retention.minutes = nulllog.retention.ms = nulllog.roll.hours = 168log.roll.jitter.hours = 0log.roll.jitter.ms = nulllog.roll.ms = nulllog.segment.bytes = 1073741824log.segment.delete.delay.ms = 60000max.connections.per.ip = 2147483647max.connections.per.ip.overrides =max.incremental.fetch.session.cache.slots = 1000message.max.bytes = 1000012metric.reporters = []metrics.num.samples = 2metrics.recording.level = INFOmetrics.sample.window.ms = 30000min.insync.replicas = 1num.io.threads = 8num.network.threads = 3num.partitions = 1num.recovery.threads.per.data.dir = 1num.replica.alter.log.dirs.threads = nullnum.replica.fetchers = 1offset.metadata.max.bytes = 4096offsets.commit.required.acks = -1offsets.commit.timeout.ms = 5000offsets.load.buffer.size = 5242880offsets.retention.check.interval.ms = 600000offsets.retention.minutes = 10080offsets.topic.compression.codec = 0offsets.topic.num.partitions = 50offsets.topic.replication.factor = 1offsets.topic.segment.bytes = 104857600password.encoder.cipher.algorithm = AES/CBC/PKCS5Paddingpassword.encoder.iterations = 4096password.encoder.key.length = 128password.encoder.keyfactory.algorithm = nullpassword.encoder.old.secret = nullpassword.encoder.secret = nullport = 9092principal.builder.class = nullproducer.purgatory.purge.interval.requests = 1000queued.max.request.bytes = -1queued.max.requests = 500quota.consumer.default = 9223372036854775807quota.producer.default = 9223372036854775807quota.window.num = 11quota.window.size.seconds = 1replica.fetch.backoff.ms = 1000replica.fetch.max.bytes = 1048576replica.fetch.min.bytes = 1replica.fetch.response.max.bytes = 10485760replica.fetch.wait.max.ms = 500replica.high.watermark.checkpoint.interval.ms = 5000replica.lag.time.max.ms = 10000replica.socket.receive.buffer.bytes = 65536replica.socket.timeout.ms = 30000replication.quota.window.num = 11replication.quota.window.size.seconds = 1request.timeout.ms = 30000reserved.broker.max.id = 1000sasl.client.callback.handler.class = nullsasl.enabled.mechanisms = [GSSAPI]sasl.jaas.config = nullsasl.kerberos.kinit.cmd = /usr/bin/kinitsasl.kerberos.min.time.before.relogin = 60000sasl.kerberos.principal.to.local.rules = [DEFAULT]sasl.kerberos.service.name = nullsasl.kerberos.ticket.renew.jitter = 0.05sasl.kerberos.ticket.renew.window.factor = 0.8sasl.login.callback.handler.class = nullsasl.login.class = nullsasl.login.refresh.buffer.seconds = 300sasl.login.refresh.min.period.seconds = 60sasl.login.refresh.window.factor = 0.8sasl.login.refresh.window.jitter = 0.05sasl.mechanism.inter.broker.protocol = GSSAPIsasl.server.callback.handler.class = nullsecurity.inter.broker.protocol = PLAINTEXTsocket.receive.buffer.bytes = 102400socket.request.max.bytes = 104857600socket.send.buffer.bytes = 102400ssl.cipher.suites = []ssl.client.auth = nonessl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]ssl.endpoint.identification.algorithm = httpsssl.key.password = nullssl.keymanager.algorithm = SunX509ssl.keystore.location = nullssl.keystore.password = nullssl.keystore.type = JKSssl.protocol = TLSssl.provider = nullssl.secure.random.implementation = nullssl.trustmanager.algorithm = PKIXssl.truststore.location = nullssl.truststore.password = nullssl.truststore.type = JKStransaction.abort.timed.out.transaction.cleanup.interval.ms = 60000transaction.max.timeout.ms = 900000transaction.remove.expired.transaction.cleanup.interval.ms = 3600000transaction.state.log.load.buffer.size = 5242880transaction.state.log.min.isr = 1transaction.state.log.num.partitions = 50transaction.state.log.replication.factor = 1transaction.state.log.segment.bytes = 104857600transactional.id.expiration.ms = 604800000unclean.leader.election.enable = falsezookeeper.connect = master:2181,node02:2181,node03:2181zookeeper.connection.timeout.ms = 6000zookeeper.max.in.flight.requests = 10zookeeper.session.timeout.ms = 6000zookeeper.set.acl = falsezookeeper.sync.time.ms = 2000(kafka.server.KafkaConfig)
[2022-04-15 13:47:13,729] INFO KafkaConfig values:advertised.host.name = nulladvertised.listeners = nulladvertised.port = nullalter.config.policy.class.name = nullalter.log.dirs.replication.quota.window.num = 11alter.log.dirs.replication.quota.window.size.seconds = 1authorizer.class.name =auto.create.topics.enable = trueauto.leader.rebalance.enable = truebackground.threads = 10broker.id = 0broker.id.generation.enable = truebroker.rack = nullclient.quota.callback.class = nullcompression.type = producerconnections.max.idle.ms = 600000controlled.shutdown.enable = truecontrolled.shutdown.max.retries = 3controlled.shutdown.retry.backoff.ms = 5000controller.socket.timeout.ms = 30000create.topic.policy.class.name = nulldefault.replication.factor = 1delegation.token.expiry.check.interval.ms = 3600000delegation.token.expiry.time.ms = 86400000delegation.token.master.key = nulldelegation.token.max.lifetime.ms = 604800000delete.records.purgatory.purge.interval.requests = 1delete.topic.enable = truefetch.purgatory.purge.interval.requests = 1000group.initial.rebalance.delay.ms = 0group.max.session.timeout.ms = 300000group.min.session.timeout.ms = 6000host.name =inter.broker.listener.name = nullinter.broker.protocol.version = 2.0-IV1leader.imbalance.check.interval.seconds = 300leader.imbalance.per.broker.percentage = 10listener.security.protocol.map = PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSLlisteners = nulllog.cleaner.backoff.ms = 15000log.cleaner.dedupe.buffer.size = 134217728log.cleaner.delete.retention.ms = 86400000log.cleaner.enable = truelog.cleaner.io.buffer.load.factor = 0.9log.cleaner.io.buffer.size = 524288log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308log.cleaner.min.cleanable.ratio = 0.5log.cleaner.min.compaction.lag.ms = 0log.cleaner.threads = 1log.cleanup.policy = [delete]log.dir = /tmp/kafka-logslog.dirs = /root/hadoop/kafka_2.11-2.0.0/kafka-logslog.flush.interval.messages = 9223372036854775807log.flush.interval.ms = nulllog.flush.offset.checkpoint.interval.ms = 60000log.flush.scheduler.interval.ms = 9223372036854775807log.flush.start.offset.checkpoint.interval.ms = 60000log.index.interval.bytes = 4096log.index.size.max.bytes = 10485760log.message.downconversion.enable = truelog.message.format.version = 2.0-IV1log.message.timestamp.difference.max.ms = 9223372036854775807log.message.timestamp.type = CreateTimelog.preallocate = falselog.retention.bytes = -1log.retention.check.interval.ms = 300000log.retention.hours = 168log.retention.minutes = nulllog.retention.ms = nulllog.roll.hours = 168log.roll.jitter.hours = 0log.roll.jitter.ms = nulllog.roll.ms = nulllog.segment.bytes = 1073741824log.segment.delete.delay.ms = 60000max.connections.per.ip = 2147483647max.connections.per.ip.overrides =max.incremental.fetch.session.cache.slots = 1000message.max.bytes = 1000012metric.reporters = []metrics.num.samples = 2metrics.recording.level = INFOmetrics.sample.window.ms = 30000min.insync.replicas = 1num.io.threads = 8num.network.threads = 3num.partitions = 1num.recovery.threads.per.data.dir = 1num.replica.alter.log.dirs.threads = nullnum.replica.fetchers = 1offset.metadata.max.bytes = 4096offsets.commit.required.acks = -1offsets.commit.timeout.ms = 5000offsets.load.buffer.size = 5242880offsets.retention.check.interval.ms = 600000offsets.retention.minutes = 10080offsets.topic.compression.codec = 0offsets.topic.num.partitions = 50offsets.topic.replication.factor = 1offsets.topic.segment.bytes = 104857600password.encoder.cipher.algorithm = AES/CBC/PKCS5Paddingpassword.encoder.iterations = 4096password.encoder.key.length = 128password.encoder.keyfactory.algorithm = nullpassword.encoder.old.secret = nullpassword.encoder.secret = nullport = 9092principal.builder.class = nullproducer.purgatory.purge.interval.requests = 1000queued.max.request.bytes = -1queued.max.requests = 500quota.consumer.default = 9223372036854775807quota.producer.default = 9223372036854775807quota.window.num = 11quota.window.size.seconds = 1replica.fetch.backoff.ms = 1000replica.fetch.max.bytes = 1048576replica.fetch.min.bytes = 1replica.fetch.response.max.bytes = 10485760replica.fetch.wait.max.ms = 500replica.high.watermark.checkpoint.interval.ms = 5000replica.lag.time.max.ms = 10000replica.socket.receive.buffer.bytes = 65536replica.socket.timeout.ms = 30000replication.quota.window.num = 11replication.quota.window.size.seconds = 1request.timeout.ms = 30000reserved.broker.max.id = 1000sasl.client.callback.handler.class = nullsasl.enabled.mechanisms = [GSSAPI]sasl.jaas.config = nullsasl.kerberos.kinit.cmd = /usr/bin/kinitsasl.kerberos.min.time.before.relogin = 60000sasl.kerberos.principal.to.local.rules = [DEFAULT]sasl.kerberos.service.name = nullsasl.kerberos.ticket.renew.jitter = 0.05sasl.kerberos.ticket.renew.window.factor = 0.8sasl.login.callback.handler.class = nullsasl.login.class = nullsasl.login.refresh.buffer.seconds = 300sasl.login.refresh.min.period.seconds = 60sasl.login.refresh.window.factor = 0.8sasl.login.refresh.window.jitter = 0.05sasl.mechanism.inter.broker.protocol = GSSAPIsasl.server.callback.handler.class = nullsecurity.inter.broker.protocol = PLAINTEXTsocket.receive.buffer.bytes = 102400socket.request.max.bytes = 104857600socket.send.buffer.bytes = 102400ssl.cipher.suites = []ssl.client.auth = nonessl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]ssl.endpoint.identification.algorithm = httpsssl.key.password = nullssl.keymanager.algorithm = SunX509ssl.keystore.location = nullssl.keystore.password = nullssl.keystore.type = JKSssl.protocol = TLSssl.provider = nullssl.secure.random.implementation = nullssl.trustmanager.algorithm = PKIXssl.truststore.location = nullssl.truststore.password = nullssl.truststore.type = JKStransaction.abort.timed.out.transaction.cleanup.interval.ms = 60000transaction.max.timeout.ms = 900000transaction.remove.expired.transaction.cleanup.interval.ms = 3600000transaction.state.log.load.buffer.size = 5242880transaction.state.log.min.isr = 1transaction.state.log.num.partitions = 50transaction.state.log.replication.factor = 1transaction.state.log.segment.bytes = 104857600transactional.id.expiration.ms = 604800000unclean.leader.election.enable = falsezookeeper.connect = master:2181,node02:2181,node03:2181zookeeper.connection.timeout.ms = 6000zookeeper.max.in.flight.requests = 10zookeeper.session.timeout.ms = 6000zookeeper.set.acl = falsezookeeper.sync.time.ms = 2000(kafka.server.KafkaConfig)
[2022-04-15 13:47:13,747] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-04-15 13:47:13,747] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-04-15 13:47:13,750] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-04-15 13:47:13,771] INFO Loading logs. (kafka.log.LogManager)
[2022-04-15 13:47:13,776] INFO Logs loading complete in 5 ms. (kafka.log.LogManager)
[2022-04-15 13:47:13,782] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-04-15 13:47:13,787] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-04-15 13:47:13,978] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2022-04-15 13:47:14,003] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2022-04-15 13:47:14,020] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,020] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,021] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,031] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2022-04-15 13:47:14,069] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2022-04-15 13:47:14,076] INFO Result of znode creation at /brokers/ids/0 is: OK (kafka.zk.KafkaZkClient)
[2022-04-15 13:47:14,078] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(client1,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2022-04-15 13:47:14,081] WARN No meta.properties file under dir /root/hadoop/kafka_2.11-2.0.0/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2022-04-15 13:47:14,127] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,130] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,131] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-04-15 13:47:14,132] INFO Creating /controller (is it secure? false) (kafka.zk.KafkaZkClient)
[2022-04-15 13:47:14,139] INFO Result of znode creation at /controller is: OK (kafka.zk.KafkaZkClient)
[2022-04-15 13:47:14,146] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2022-04-15 13:47:14,146] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2022-04-15 13:47:14,148] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 2 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2022-04-15 13:47:14,165] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2022-04-15 13:47:14,186] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-04-15 13:47:14,187] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2022-04-15 13:47:14,187] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2022-04-15 13:47:14,234] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2022-04-15 13:47:14,251] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2022-04-15 13:47:14,253] INFO Kafka version : 2.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-15 13:47:14,253] INFO Kafka commitId : 3402a8361b734732 (org.apache.kafka.common.utils.AppInfoParser)
[2022-04-15 13:47:14,254] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

再一个终端运行。

root@client1:~# kafka-topics.sh --create --zookeeper master:2181,node02:2181,node03:2181 --replication-factor 1 --partitions 1 --topic dblab
Created topic "dblab".
root@client1:~# kafka-topics.sh --list --zookeeper master:2181,node02:2181,node03:2181
dblab
root@client1:~# kafka-console-producer.sh --broker-list localhost:9092 --topic dblab
>hello hadoop
>hello xmu
>hello world
>

再个终端看到结果。

root@client1:~# kafka-console-consumer.sh --topic dblab --from-beginning --bootstrap-server localhost:9092
hello hadoop
hello xmu
hello world

4 Flink

将hadoop支持文件复制到flink lib中。

root@client1:~/data# cp flink-shaded-hadoop-2-uber-2.7.5-10.0.jar /root/hadoop/flink-1.10.2/lib
root@client1:~/data# scp flink-shaded-hadoop-2-uber-2.7.5-10.0.jar root@master:/root/hadoop/flink-1.10.2/lib
root@master's password:
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar                                             100%   39MB 206.0MB/s   00:00
root@client1:~/data# scp flink-shaded-hadoop-2-uber-2.7.5-10.0.jar root@node02:/root/hadoop/flink-1.10.2/lib
root@node02's password:
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar                                             100%   39MB 205.2MB/s   00:00
root@client1:~/data# scp flink-shaded-hadoop-2-uber-2.7.5-10.0.jar root@node03:/root/hadoop/flink-1.10.2/lib
root@node03's password:
flink-shaded-hadoop-2-uber-2.7.5-10.0.jar                                             100%   39MB 201.8MB/s   00:00

master节点启动flink集群。

[root@master ~]# start-cluster.sh
Starting HA cluster with 1 masters.
Starting standalonesession daemon on host master.
Starting taskexecutor daemon on host node02.
Starting taskexecutor daemon on host node03.
[root@master ~]# jps
3778 StandaloneSessionClusterEntrypoint
1891 NameNode
2299 ResourceManager
3867 Jps
782 QuorumPeerMain
2110 SecondaryNameNode
[root@master ~]#

运行flink自带样例。

root@client1:~/data# flink run /root/hadoop/flink-1.10.2/examples/batch/WordCount.jar
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
Job has been submitted with JobID edd976bb874ab5a5030d2e79f5d3fa1f
Program execution finished
Job with JobID edd976bb874ab5a5030d2e79f5d3fa1f has finished.
Job Runtime: 617 ms
Accumulator Results:
- 9a7c97a7030473900ef75f68bda2e78f (java.util.ArrayList) [170 elements](a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
(bodkin,1)
(bourn,1)
...

运行flink shell。

root@client1:~# start-scala-shell.sh local
Starting Flink Shell:Starting local Flink cluster (host: localhost, port: 8081).Connecting to Flink cluster (host: localhost, port: 8081).???????????????????????????????        ???????  ?????   ?????????      ????????         ???????    ????????            ???   ???????       ????????????????? ?   ???       ?????? ??????????   ????      ????? ????????????       ???    ??????? ???????????? ??         ??    ??????????????????  ??           ?   ?? ???????????  ???            ?  ?? ???????? ????????? ? ??          ? ?? ????????    ????  ?????? ????          ??????????       ??? ?? ???????? ?? ???       ???????????         ????  ? ?  ??????  ?? ??? ?????????              ????           ?????    ? ???????              ????????          ??? ?????    ???    ????????????????????            ????  ?
????? ???   ??????   ????????                  ????  ??
????????  ???????????????                            ??
?? ????   ???????  ???       ??????    ??          ???
??? ???  ???  ???????            ????   ???????????????? ?????  ????  ??                ??      ????   ?????   ???   ?     ??                ??              ????   ??         ??                 ??        ?????????? ?????       ??                  ???????????    ????   ????      ?                    ???????      ?????   ?????                         ?? ???????????????    ????                     ??????? ?????????????                          ??  ????  ??????????????????????????????????????  ?????F L I N K - S C A L A - S H E L LNOTE: Use the prebound Execution Environments and Table Environment to implement batch or streaming programs.Batch - Use the 'benv' and 'btenv' variable* val dataSet = benv.readTextFile("/path/to/data")* dataSet.writeAsText("/path/to/output")* benv.execute("My batch program")** val batchTable = btenv.fromDataSet(dataSet)* btenv.registerTable("tableName", batchTable)* val result = btenv.sqlQuery("SELECT * FROM tableName").collectHINT: You can use print() on a DataSet to print the contents or collect()a sql query result back to the shell.Streaming - Use the 'senv' and 'stenv' variable* val dataStream = senv.fromElements(1, 2, 3, 4)* dataStream.countWindowAll(2).sum(0).print()** val streamTable = stenv.fromDataStream(dataStream, 'num)* val resultTable = streamTable.select('num).where('num % 2 === 1 )* resultTable.toAppendStream[Row].print()* senv.execute("My streaming program")HINT: You can only print a DataStream to the shell in local mode.scala> 8*2+5
res0: Int = 21scala>

修改maven settings.xml，使用阿里源。

root@client1:~/apache-maven-3.8.5/conf# vi settings.xml

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0http://maven.apache.org/xsd/settings-1.0.0.xsd"><mirrors><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云公共仓库</name><url>https://maven.aliyun.com/repository/public</url></mirror><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云谷歌仓库</name><url>https://maven.aliyun.com/repository/google</url></mirror><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云阿帕奇仓库</name><url>https://maven.aliyun.com/repository/apache-snapshots</url></mirror><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云spring仓库</name><url>https://maven.aliyun.com/repository/spring</url></mirror><mirror><id>aliyunmaven</id><mirrorOf>*</mirrorOf><name>阿里云spring插件仓库</name><url>https://maven.aliyun.com/repository/spring-plugin</url></mirror></mirrors>
</settings>

idea新建项目FlinkWordCount。

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>org.example</groupId><artifactId>FlinkWordCount</artifactId><version>1.0-SNAPSHOT</version><properties><maven.compiler.source>8</maven.compiler.source><maven.compiler.target>8</maven.compiler.target></properties><repositories><repository><id>alimaven</id><name>aliyun maven</name><url>http://maven.aliyun.com/nexus/content/groups/public/</url></repository></repositories><dependencies><dependency><groupId>org.apache.flink</groupId><artifactId>flink-scala_2.11</artifactId><version>1.10.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-scala_2.11</artifactId><version>1.10.2</version></dependency><dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients_2.11</artifactId><version>1.10.2</version></dependency></dependencies><build><plugins><plugin><groupId>net.alchim31.maven</groupId><artifactId>scala-maven-plugin</artifactId><version>3.4.6</version><executions><execution><goals><goal>compile</goal></goals></execution></executions></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-assembly-plugin</artifactId><version>3.0.0</version><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>
</project>

WordCount.scala

import org.apache.flink.api.scala._object WordCount {def main(args: Array[String]): Unit = {//第1步：建立执行环境val env = ExecutionEnvironment.getExecutionEnvironment//第2步：创建数据源val text = env.fromElements("hello, world!","hello, world!","hello, world!")//第3步：对数据集指定转换操作val counts = text.flatMap { _.toLowerCase.split(" ") }.map { (_, 1) }.groupBy(0).sum(1)// 第4步：输出结果counts.print()}
}

运行结果

idea新建项目FlinkStreamWordCount。

WordCount.scala

import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.windowing.time.Timeobject StreamWordCount{def main(args: Array[String]): Unit = {//第1步：建立执行环境val env = StreamExecutionEnvironment.getExecutionEnvironment//第2步：创建数据源val source = env.socketTextStream("localhost",9999,'\n')//第3步：对数据集指定转换操作逻辑val dataStream = source.flatMap(_.split(" ")).map((_,1)).keyBy(0).timeWindow(Time.seconds(2),Time.seconds(2)).sum(1)//第4步：指定计算结果输出位置dataStream.print()//第5步：指定名称并触发流计算env.execute("Flink Streaming Word Count")}
}

打包。

开启终端运行nc。

root@client1:~# cd
root@client1:~# nc -lk 9999

另终端运行WordCount。

root@client1:~# flink run --class StreamWordCount /root/devz/FlinkStreamWordCount/out/artifacts/FlinkStreamWordCount_jar/FlinkStreamWordCount.jar

5 Redis

docker hub拉取redis:4.0。

sudo docker pull redis:4.0

启动redis容器。

sudo docker run -d --name rediss -p 6379:6379 redis:4.0

使用Another Redis Desktop Manager连接。

Windows PC上创建大数据职业技能竞赛实验环境之六--Flume、Kafka和Flink编程相关推荐

Windows PC上创建大数据职业技能竞赛实验环境之四--客户端环境的搭建
Hadoop集群访问客户端 X window Jet brain Idea和客户端开发环境以上环境我们将在WSL的Ubuntu docker容器中实现,其中对X和Idea的访问,通过Windows ...
Windows PC上创建大数据职业技能竞赛实验环境之一--基本环境的搭建
1 大数据职业竞赛环境的基础要求软件环境设备类型软件类别软件名称.版本号竞赛服务器竞赛环境大数据集群操作系统 CentOS 7 大数据平台组件 Hadoop 2.7.7 Hive 2.3. ...
Windows PC上创建大数据职业技能竞赛实验环境之五--hadoop、hive和spark编程
前述,我们在Windows WSL上,通过Docker容器技术实现了Hadoop集群环境,现在看看利用现有集群环境进行编程开发. 1 设置容器开启时自运行SSH服务参照docker中安装Ubuntu ...
Windows PC上创建大数据职业技能竞赛实验环境之七--疑难问题
1 Ubuntu Docker中文显示问题查看locale设置,我们可以看到Ubuntu缺省local是POSIX,需要进行修改. root@client1:~# locale LANG= LANG ...
在Kaggle上赢得大数据竞赛的技巧和窍门
在Kaggle上赢得大数据竞赛的技巧和窍门解决方案平台数据应用方法阅读1906 原文:The tips and tricks I used to succeed on Kaggle 作 ...
就业培训 | 2020第一期重庆高校毕业生大数据职业技能线上特训营开课啦
"大数据特训营开课啦!""芝诺数据的老师太'可'了~"这几天,同学们的朋友圈被刷屏了,原来,是2020第一期重庆高校毕业生大数据职业技能线上特训营开讲了! &q ...
全国大学生大数据技能竞赛（Hadoop集群搭建）
系列文章全国大学生大数据技能竞赛(数仓部署) 全国大学生大数据技能竞赛(Spark on Yarn安装) 文章目录系列文章前言资料链接用VMware练习配置前准备三台虚拟机并修改网络为桥接 ...
高校大数据专业竞赛建设方案
第一章建设背景 1.1 政策分析  2017年1月工业和信息化部正式发布了<大数据产业发展规划(2016-2020年)>,明确了"十三五"时期大数据产业的发展思路 ...
oracle移花接木,移植苹果，Windows PC上安装MacOS
苹果系统作为一款优秀的操作系统,仅仅体验外观我们当然无法真正感受到苹果系统的魅力.虽然大部分朋友没有苹果电脑,不过现在我们可以在Windows PC上安装实际苹果系统.由于苹果电脑和Windows差异 ...

Windows PC上创建大数据职业技能竞赛实验环境之六--Flume、Kafka和Flink编程

Windows PC上创建大数据职业技能竞赛实验环境之六--Flume、Kafka和Flink编程相关推荐

最新文章

热门文章