Flume环境搭建_五种案例(转)
Flume环境搭建_五种案例
http://flume.apache.org/FlumeUserGuide.html
A simple example
Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.
# example.conf: A single-node Flume configuration# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.
Given this configuration file, we can start Flume as follows:
$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Note that in a full deployment we would typically include one more option: --conf=<conf-dir>. The <conf-dir> directory would include a shell script flume-env.sh and potentially a log4j properties file. In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.
From a separate terminal, we can then telnet port 44444 and send Flume an event:
$ telnet localhost 44444 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. Hello world! <ENTER> OK
The original Flume terminal will output the event in a log message.
12/06/19 15:32:19 INFO source.NetcatSource: Source starting 12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. }
Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.
以下为具体搭建流程
Flume搭建_案例一:单个Flume
安装node2上
1. 上传到/home/tools,解压,解压后移动到/home下
2. 重命名,并修改flume-env.sh
Flume搭建_案例二:两个Flume做集群
MemoryChanel配置
capacity:默认该通道中最大的可以存储的event数量是100,
trasactionCapacity:每次最大可以从source中拿到或者送到sink中的event数量也是100
keep-alive:event添加到通道中或者移出的允许时间
byte**:即event的字节量的限制,只包括eventbody
在/etc/profile下
node1,node2下创建测试目录test_flume,并分别在node1,node2下创建配置文件——flume21,flume22
先启动node02的Flume
flume-ng agent -n a1 -c conf -f avro.conf -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume22 -Dflume.root.logger=INFO,console
再启动node01的Flume
flume-ng agent -n a1 -c conf -f simple.conf2 -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume21 -Dflume.root.logger=INFO,console
Flume搭建_案例三:如何监控一个文件的变化?
在/etc/profile下
启动Flume
flume-ng agent -n a1 -c conf -f exec.conf -Dflume.root.logger=INFO,console
flume-ng agent -n a1 -c conf -f /home/test_flume/flume3 -Dflume.root.logger=INFO,console
Flume搭建_案例四: 如何监控一个文件:目录的变化?
在/etc/profile下
Flume搭建_案例五: 如何定义一个HDFS类型的Sink?
Flume搭建_案例五_配置项解读
hdfs.rollInterval | 30 | Number of seconds to wait before rolling current file (0 = never roll based on time interval) |
hdfs.rollSize | 1024 | File size to trigger roll, in bytes (0: never roll based on file size) |
hdfs.rollCount | 10 | Number of events written to file before it rolled (0 = never roll based on number of events) |
4. 多长时间没有操作,Flume将一个临时文件生成新文件?
hdfs.idleTimeout | 0 | Timeout after which inactive files get closed (0 = disable automatic closing of idle files) |
5. 多长时间生成一个新的目录?(比如每10s生成一个新的目录)
四舍五入,没有五入,只有四舍
(比如57分划分为55分,5,6,7,8,9在一个目录,10,11,12,13,14在一个目录)
hdfs.round | false | Should the timestamp be rounded down (if true, affects all time based escape sequences except %t) |
hdfs.roundValue | 1 | Rounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than current time. |
hdfs.roundUnit | second | The unit of the round down value - second, minute or hour. |
在/etc/profile下
转载于:https://www.cnblogs.com/wangbin/p/8192950.html
Flume环境搭建_五种案例(转)相关推荐
- 高通Android智能平台环境搭建_编译流程分析
高通Android智能平台环境搭建_编译流程分析 高通平台环境搭建,编译,系统引导流程分析 TOC \o \h \z \u 1. 高通平台android开发总结. 7 1.1 搭建高通平台环境开发环境 ...
- 代驾APP_第一章_项目环境搭建_第二节
代驾APP_第一章_项目环境搭建_第二节 文章目录 代驾APP_第一章_项目环境搭建_第二节 1-11 创建bff-driver服务 一.创建项目 二.配置pom.xml文件 三.编写YML配置文件 ...
- OpenStack环境搭建(五:附加项虚拟机文件备份使用)
实验要求: 完成Virtual box平台安装,会应用相关操作: 在virtual box虚拟平台上部署Fuel Master节点: 在virtual box虚拟平台上部署计算节点Computer: ...
- ESXi6.5环境搭建(五:常见问题及解决方案实验总结)
实验目的及要求 完成VMware workstations安装,会应用相关操作: 完成虚拟机中ESXI6.5平台的安装及网络环境配置: 完成VMware vSphere Client 6.0软件在PC ...
- 从环境搭建到回归神经网络案例,带你掌握Keras
摘要:Keras作为神经网络的高级包,能够快速搭建神经网络,它的兼容性非常广,兼容了TensorFlow和Theano. 本文分享自华为云社区<[Python人工智能] 十六.Keras环境搭建 ...
- go环境搭建_容器化 Go 开发环境的尝试
Go语言中文网,致力于每日分享编码知识,欢迎关注我,会有意想不到的收获! 01 写在前面 昨天上午"起早"到附近的影院看<波西米亚狂想曲>,选的座位比较靠前,最后的二十 ...
- 代驾APP_第一章_项目环境搭建_第一节
代驾APP_第一章_项目环境搭建 文章目录 代驾APP_第一章_项目环境搭建 1-1 创建根项目 一.创建SpringBoot项目 二.覆盖pom.xml文件 1-2 创建根项目 一.创建公共模块 二 ...
- python服务器环境搭建_服务器python环境搭建
广告关闭 腾讯云11.11云上盛惠 ,精选热门产品助力上云,云服务器首年88元起,买的越多返的越多,最高返5000元! python搭建服务器以快捷著称,实际上,我们也可以使用python搭建简易的服 ...
- java后台环境搭建_后端-java环境搭建
java环境搭建 开发工具:idea/eclipse/myeclipse + mysql/oracle/sqlserver/postgresql + tomcat6/7/8/9. 下面以eclipse ...
最新文章
- 跟大家聊聊我们为什么要学习源码?学习源码对我们有用吗?(源码感悟)
- 代码片段管理器——SnippetsLab
- goland 远程调试 golang
- 程序员随笔:使用来自服务器的图像(有源码)
- python threading模块的方法_Python THREADING模块中的JOIN()方法深入理解
- 生产订单结算KKS1常见错误
- 【Weiss】【第03章】练习3.7:有序多项式相乘
- 复杂对象的组装与创建——建造者模式
- MySQL8.0与MySQL5.7 OLTP 性能测试对比
- PULSE:一种基于隐式空间的图像超分辨率算法
- linux常用的文件操作命令大全,(办公)记事本_Linux常用的文件操作命令
- 大学英语 计算机应用基础,计算机应用基础-大学英语b.doc
- go语言之进阶篇面向对象编程
- 检测相关问题面试准备
- linux中oracle数据乱码,Linux环境解决Oracle 中文乱码
- 免下载券下载文档的方法
- 使用go进行人脸识别活体检测
- 说说怎么保证线程安全
- Ubuntu 部署 Flask + WSGI + Nginx 详解
- Hive中自定义函数详解
热门文章
- 论文阅读(3)--SPDA-CNN: Unifying Semantic Part Detection and Abstraction for Fine-grained Recognition
- PrefixSpan算法原理总结
- 关于eclipse项目的x号报错的一些问题
- idea首次创建新模块的详细操作
- Django框架(十二)-- Djang与Ajax
- String类的一些常见的获取方法(5)
- ※交换排序(1)——快速排序(quick sort)
- 设计模式_第二篇_策略模式
- POI导出EXCEL设置高度和宽度
- vue router-link 添加点击事件