channels

–

本source要发往的channel

type

–

本source的类别名称：exec

command

–

本source所要运行的linux命令,比如： tail -F /path/file

shell

–

指定运行上述命令所用shell

restartThrottle

10000

命令die了以后，重启的时间间隔

restart

false

命令die了以后，是否要重启

logStdErr

false

是否收集命令的错误输出stderr

batchSize

20

提交的event批次大小

batchTimeout

3000

发往下游没完成前，等待的时间

selector.type

replicating

指定channel选择器：replicating or multiplexing

selector.*

选择器的具体参数

interceptors

–

指定拦截器

interceptors.*

指定的拦截器的具体参数

Property Name

Default

Description

channels

–

type

–

The component type name, needs to be spooldir.

spoolDir

–

The directory from which to read files from.

fileSuffix

.COMPLETED

采集完成的文件，添加什么后缀名

deletePolicy

never

是否删除采完的文件: never or immediate

fileHeader

false

是否将所采集文件的绝对路径添加到header中

fileHeaderKey

file

上述header的key名称

basenameHeader

false

是否将文件名添加到header

basenameHeaderKey

basename

上述header的key名称

includePattern

^.*$

指定需要采集的文件名的正则表达式

ignorePattern

^$

指定要排除的文件名的正则表达式

如果一个文件名即符合includePattern又匹配ignorePattern，则该文件不采

trackerDir

.flumespool

记录元数据的目录所在路径，可以用绝对路径也可以用相对路径（相对于采集目录）

trackingPolicy

rename

采集进度跟踪策略，有两种： “rename”和 “tracker_dir”. 本参数只在deletePolicy=never时才生效

“rename”- 采完的文件根据filesuffix重命名

“tracker_dir” - 采完的文件会在trackerDir目录中生成一个同名的空文件

consumeOrder

oldest

采集顺序： oldest, youngest and random.

oldest和youngest情况下，可能会带来一定效率的损失；（需要对文件夹中所有文件进行一次扫描以寻找最old或最young的）

pollDelay

500

Delay (in milliseconds) used when polling for new files.

recursiveDirectorySearch

false

Whether to monitor sub directories for new files to read.

maxBackoff

4000

The maximum time (in millis) to wait between consecutive attempts to write to the channel(s) if the channel is full. The source will start at a low backoff and increase it exponentially each time the channel throws a ChannelException, upto the value specified by this parameter.

batchSize

100

一次传输到channel的event条数（一批）

inputCharset

UTF-8

Character set used by deserializers that treat the input file as text.

decodeErrorPolicy

FAIL

What to do when we see a non-decodable character in the input file. FAIL: Throw an exception and fail to parse the file. REPLACE: Replace the unparseable character with the “replacement character” char, typically Unicode U+FFFD. IGNORE: Drop the unparseable character sequence.

deserializer

LINE

Specify the deserializer used to parse the file into events. Defaults to parsing each line as an event. The class specified must implementEventDeserializer.Builder.

deserializer.*

Varies per event deserializer.

bufferMaxLines

–

(Obselete) This option is now ignored.

bufferMaxLineLength

5000

(Deprecated) Maximum length of a line in the commit buffer. Use deserializer.maxLineLength instead.

selector.type

replicating

replicating or multiplexing

selector.*

Depends on the selector.type value

interceptors

–

Space-separated list of interceptors

interceptors.*

Property Name

Default

Description

channels

–

type

–

本source的别名： avro

bind

–

要绑定的地址

port

–

要绑定的端口号

threads

–

服务的最大线程数

selector.type

selector.*

interceptors

–

Space-separated list of interceptors

interceptors.*

compression-type

none

压缩类型：跟发过来的数据是否压缩要匹配：none | deflate

ssl

false

Set this to true to enable SSL encryption. If SSL is enabled, you must also specify a “keystore” and a “keystore-password”, either through component level parameters (see below) or as global SSL parameters (see SSL/TLS support section).

keystore

–

This is the path to a Java keystore file. If not specified here, then the global keystore will be used (if defined, otherwise configuration error).

keystore-password

–

The password for the Java keystore. If not specified here, then the global keystore password will be used (if defined, otherwise configuration error).

keystore-type

JKS

The type of the Java keystore. This can be “JKS” or “PKCS12”. If not specified here, then the global keystore type will be used (if defined, otherwise the default is JKS).

exclude-protocols

SSLv3

Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols specified.

include-protocols

–

Space-separated list of SSL/TLS protocols to include. The enabled protocols will be the included protocols without the excluded protocols. If included-protocols is empty, it includes every supported protocols.

exclude-cipher-suites

–

Space-separated list of cipher suites to exclude.

include-cipher-suites

–

Space-separated list of cipher suites to include. The enabled cipher suites will be the included cipher suites without the excluded cipher suites. If included-cipher-suites is empty, it includes every supported cipher suites.

ipFilter

false

Set this to true to enable ipFiltering for netty

ipFilterRules

–

Define N netty ipFilter pattern rules with this config.

while true

do

echo "hello "  >> /tmp/logs/access.log

sleep 0.1

done

Property Name

Default

Description

channels

–

所要写往的channel

type

–

本source的别名： TAILDIR.

filegroups

–

空格分割的组名，每一组代表着一批文件

g1 g2

filegroups.<filegroupName>

–

每个文件组的绝路路径，文件名可用正则表达式

positionFile

~/.flume/taildir_position.json

记录偏移量位置的文件所在路径

headers.<filegroupName>.<headerKey>

–

Header value which is the set with header key. Multiple headers can be specified for one file group.

byteOffsetHeader

false

Whether to add the byte offset of a tailed line to a header called ‘byteoffset’.

skipToEnd

false

Whether to skip the position to EOF in the case of files not written on the position file.

idleTimeout

120000

关闭非活动文件的时延。如果被关闭的这个文件又在某个时间有了新增行,会被此source检测到，并重新打开

writePosInterval

3000

3s 记录一次偏移量到positionfile

batchSize

100

提交event到channel的批次最大条数

maxBatchCount

Long.MAX_VALUE

控制在一个文件上连续读取的最大批次个数（如果某个文件正在被高速写入，那就应该让这个参数调为最大值，以让source可以集中精力专采这个文件）

backoffSleepIncrement

1000

The increment for time delay before reattempting to poll for new data, when the last attempt did not find any new data.

maxBackoffSleep

5000

The max time delay between each reattempt to poll for new data, when the last attempt did not find any new data.

cachePatternMatching

true

Listing directories and applying the filename regex pattern may be time consuming for directories containing thousands of files. Caching the list of matching files can improve performance. The order in which files are consumed will also be cached. Requires that the file system keeps track of modification times with at least a 1-second granularity.

fileHeader

false

Whether to add a header storing the absolute path filename.

fileHeaderKey

file

Header key to use when appending absolute path filename to event header.

Property Name

Default

Description

channels

–

数据发往的channel

type

–

本source的名称：

org.apache.flume.source.kafka.KafkaSource

kafka.bootstrap.servers

–

Kafka broker服务器列表，逗号分隔

kafka.consumer.group.id

flume

Kafka消费者组id

kafka.topics

–

Kafka消息主题列表，逗号隔开

kafka.topics.regex

–

用正则表达式来指定一批topic；本参数的优先级高于kafka.topics

batchSize

1000

写入channel的event 批，最大消息条数

batchDurationMillis

1000

批次写入channel的最大时长

backoffSleepIncrement

1000

Kafka Topic 显示为空时触发的初始和增量等待时间。

maxBackoffSleep

5000

Kafka Topic 显示为空时触发的最长等待时间

useFlumeEventFormat

false

默认情况下，event 将从Kafka Topic 直接作为字节直接进入event 主体。设置为true以读取event 作为Flume Avro二进制格式。与Kafka Sink上的相同属性或Kafka Channel上的parseAsFlumeEvent属性一起使用时，这将保留在生成端发送的任何Flume标头。

setTopicHeader

true

是否要往header中加入一个kv：topic信息

topicHeader

topic

应上面开关的需求，加入kv：topic =>topic名称

kafka.consumer.security.protocol

PLAINTEXT

Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.

more consumer security props

If using SASL_PLAINTEXT, SASL_SSL or SSL refer to Kafka security for additional properties that need to be set on consumer.

Other Kafka Consumer Properties

–

本source，允许直接配置任意的kafka消费者参数，格式如下：

For example: kafka.consumer.auto.offset.reset

（就是在消费者参数前加统一前缀： kafka.consumer.）