9-4push方式整合Spark streaming.

进入:/home/hadoop/app/apache-flume-1.6.0-cdh5.7.0-bin/conf

新建文件夹并修改它

[hadoop@hadoop000 conf]$ vi flume_push_streaming.conf

# exec-memory-avro.conf
simple-agent.sources = netcat-source
simple-agent.sinks = avro-sink
simple-agent.channels = memory-channel# Describe/configure the source
simple-agent.sources.netcat-source.type = netcat
simple-agent.sources.netcat-source.bind = hadoop000
simple-agent.sources.netcat-source.port = 44444# Describe the sink
simple-agent.sinks.avro-sink.type = avro
simple-agent.sinks.avro-sink.hostname = hadoop000
simple-agent.sinks.avro-sink.port = 41414# Use a channel which buffers events in memory
simple-agent.channels.memory-channel.type = memory# Bind the source and sink to the channel
simple-agent.sources.netcat-source.channels = memory-channel
simple-agent.sinks.avro-sink.channel = memory-channel

9-4push方式整合Spark streaming.相关推荐

Scala和Java二种方式实战Spark Streaming开发
一.Java方式开发 1.开发前准备:假定您以搭建好了Spark集群. 2.开发环境采用eclipse maven工程,需要添加Spark Streaming依赖. 3.Spark streaming ...
Spark Streaming读取Kafka数据的两种方式
Kafka在0.8和0.10之间引入了一种新的消费者API,因此,Spark Streaming与Kafka集成,有两种包可以选择: spark-streaming-kafka-0-8与spark-s ...
sparkstreaming监听hdfs目录_大数据系列之Spark Streaming接入Kafka数据
Spark Streaming官方提供Receiver-based和Direct Approach两种方法接入Kafka数据,本文简单介绍两种方式的pyspark实现. 1.Spark Streami ...
Spark Streaming 新手指南（原始文章已经发布表在IBM Developworks）
插个小广告:本人的<大话Java性能优化>一书已经在亚马逊.当当.京东.天猫出售,提前谢谢大家的支持. 亚马逊地址:https://www.amazon.cn/%E5%A4%A7%E8%A ...
Spark Streaming 2.2.1 处理Kafka数据源的实战准备
Spark Streaming 2.2.1 处理Kafka数据源的实战准备 Kafka是一种高吞吐量的分布式发布订阅消息系统,Spark Streaming读取Kafka数据支持二种方式:Receiv ...
Spark Streaming整合flume实战
Spark Streaming对接Flume有两种方式 Poll:Spark Streaming从flume 中拉取数据 Push:Flume将消息Push推给Spark Streaming 1.安装 ...
Flume整合SparkStreaming出现UnsupportedClassVersionError: org/apache/spark/streaming/flume/sink/SparkFlum
Flume整合SparkStreaming使用Poll方式拉取数据出现 UnsupportedClassVersionError: org/apache/spark/streaming/flume/s ...
spark streaming 整合kafka 报错 KafkaConsumer is not safe for multi-threaded access
问题描述 spark streaming 使用直连方式读取kafka 数据,使用窗口时出现 java.util.ConcurrentModificationException: KafkaCons ...
DStream实战之Spark Streaming整合fulme实战, Flume向Spark Streaming中push推数据 36
前言本文所需要的安装包&Flume配置文件,博主都已上传,链接为本文涉及安装包&Flume配置文件本文涉及的安装包&Flume配置文件,请自行下载~ flume作为日志实时采 ...

9-4push方式整合Spark streaming.

9-4push方式整合Spark streaming.相关推荐

最新文章

热门文章