配置文件：
编辑 flume-conf.properties

agent1表示代理名称

agent1.sources=source1
agent1.sinks=sink1
agent1.channels=channel1

配置source1

agent1.sources.source1.type=spooldir
agent1.sources.source1.spoolDir=/usr/local/flume_logs
agent1.sources.source1.channels=channel1
agent1.sources.source1.fileHeader = false
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = timestamp

配置channel1

agent1.channels.channel1.type=file
agent1.channels.channel1.checkpointDir=/usr/local/flume_logs_tmp_cp
agent1.channels.channel1.dataDirs=/usr/local/flume_logs_tmp

配置sink1

agent1.sinks.sink1.type=hdfs
agent1.sinks.sink1.hdfs.path=hdfs://Spark1:9000/flume_logs
agent1.sinks.sink1.hdfs.fileType=DataStream
agent1.sinks.sink1.hdfs.writeFormat=TEXT
agent1.sinks.sink1.hdfs.rollInterval=1
agent1.sinks.sink1.channel=channel1
agent1.sinks.sink1.hdfs.filePrefix=%Y-%m-%d

开启：切换到Flume的bin目录下
./flume-ng agent -n agent1 -c conf -f /root/app/flume-1.5.0/conf/flume-conf.properties -Dflume.root.logger=DEBUG,consple

运行：
18/06/11 22:41:24 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:24 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:25 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:25 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:26 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:26 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:27 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:27 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:28 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:28 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:29 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:29 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.
18/06/11 22:41:30 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown.

完成！

flume的配置和开启相关推荐

大数据——Flume安装配置和使用
Flume安装配置和使用 Flume安装和配置测试监控端口数据监控文件信息监控文件夹中的新文件监控文件写入HDFS中监控文件上传到HDFS并去除首条字段信息(拦截器) 内置拦截器自定义拦 ...
linux apache gzip压缩,Linux入门教程:配置Apache开启gzip压缩传输,gzip压缩 LoadModul
Linux入门教程:配置Apache开启gzip压缩传输,gzip压缩 LoadModul 开启模块打开httpd.conf后,先将下面两行配置前面的#号去掉,这样apache就会启用这两个模块,其 ...
linux--redis的安装和配置和开启多个端口
目录一.Redis在Windows下的部署及开启多个端口服务一.以默认端口启动和安装服务器二.开启多个端口服务器二.linux--redis的安装和配置和开启多个端口方法一 (安装) 方法 ...
Win10下安转Elasticsearch6.8.6与配置安全策略-开启密码账号访问
Win10下安转Elasticsearch6.8.6与配置安全策略-开启密码账号访问一.Elasticsearch是什么? 二.下载 Elasticsearch 6.8.6下载地址三.安装四.配 ...
Flume之——配置多个Sink源（一个Source对应多个Channel和Sink）
转载请注明出处:https://blog.csdn.net/l1028386804/article/details/98055100 配置模型如下图: Flume的配置如下: myagent.sour ...
CentOS 7下 VNC 服务的配置和开启、常见问题
文章目录 1. X server.Xvnc 和 VNC server X Window System X Server X Client X Window Manager Display Manage ...
linux防火墙配置80端口,Linux配置防火墙开启80端口
Linux配置防火墙开启80端口的方法命令行输入: vi /etc/sysconfig/iptables 将 -A INPUT -m state --state NEW -m tcp -p tcp ...
kali中网卡、ssh、apache的配置与开启
kali中网卡.ssh.apache的配置与开启在Kali-linux中修改网卡文件,启动ssh和apache服务的方法 1.su root //取得root权限 2.shift+字母 //大小写字 ...
groovy安装配置及其开启indy、@CompileStatic
groovy安装配置及其开启indy.@CompileStatic 前提 groovy安装 indy版本切换 java.非indy.indy.@CompileStatic四种方式的字节码比较知识点 ...

flume的配置和开启

agent1表示代理名称

配置source1

配置channel1

配置sink1

flume的配置和开启相关推荐

最新文章

热门文章