flume报错：Space for commit to queue couldn‘t be acquired. Sinks are likely not keeping up with sources

1.问题描述

自己采用flume监控log文件，结果报错

2018-11-26 19:32:04,687 (pool-5-thread-1) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:353)] Failed while running command: tail -F /opt/datas/streamingproject/access.log
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memory-channel}at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:253)at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:191)at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:382)at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:342)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.ChannelFullException: Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the buffer size is too tightat org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:130)at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:245)... 8 more
2018-11-26 19:32:04,690 (timedFlushExecService19-0) [ERROR - org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:328)] Exception occured when processing event batch
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: memory-channel}at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:253)at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:191)at org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:382)at org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:255)at org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:324)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.ChannelException: java.lang.InterruptedExceptionat org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:154)at org.apache.flume.channel.ChannelProcessor.executeChannelTransaction(ChannelProcessor.java:245)... 11 more
Caused by: java.lang.InterruptedExceptionat java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)... 12 more
2018-11-26 19:32:04,690 (pool-5-thread-1) [INFO - org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:376)] Command [tail -F /opt/datas/streamingproject/access.log] exited with 143

2.原因

参看Space for commit to queue couldn't be acquired. Sinks are likely not keeping up with sources, or the，是由于memory channel导致的。对于内存通道，当输入过多，而无法全部输出的时候，会导致内存通道崩溃，从而报错。

3.解决

将内存通道换为file通道。生产中多用file通道，安全稳定。但是缺点是慢。

flume报错：Space for commit to queue couldn‘t be acquired. Sinks are likely not keeping up with sources相关推荐

Flume报错：Space for commit to queue couldn't be acquired Sinks are likely not keeping
今天在做Flume项目时,遇到一个错误 Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't ...
Flume报错：Sinks are likely not keeping up with sources, or the buffer size is too tight
一.问题描述 flume监控日志,同时,发送到flume监控的avro端口,当大量数据1000万条数据适合,flume监控日志的报错.然后,查看flume监控界面,发现flume监控界面消费突然消失. ...
flume报错WARN clients.NetworkClient: Error while fetching metadata with correlation id
项目场景: 修改linux系统时间,模拟使用flume读取日志文件,并把日志文件信息传输给kafka消费,kafka消费之后,再使用flume读取kafka消费后的日志信息, 问题描述: flume在 ...
rabbitmq报错：PRECONDITION_FAILED - parameters for queue ‘test-1‘ in vhost ‘/‘ not equivalent
错误如下: [root@master2 rabbitmq-python]# python send-1.py Traceback (most recent call last):File " ...
repo upload 或者git push 报错“remote: (W):commit subject 65 characters,use shorter first paragraph”
遇到问题: Repo upload pre-hook check report: Git (ambarella/include): Pass Git (kernel/l ...
git报错：git commit命令后显示Author identity unknown的解决方法
一.报错截图使用git commit命令之后, 报错:Author identity unknown 二.报错原因没有配置用户信息或者配置的用户信息已过期三.解决方法重新配置用户信息先输入: ...
启动flume报错guava版本不一致
一.问题描述启动flume(1.9) 时报错,如图 2020-12-23 15:37:58,284 ERROR hdfs.HDFSEventSink: process failed java.lan ...
git push 报错 ---error: commit 8aa8e31: email address yin.jiaqing@.....com is not registered in your
git push 报错: error: commit 8aa8e31: email address yin.jiaqing@haustart.com is not registered in your ...
svn提交报错，Error running context: 远程主机强迫关闭了一个现有的连接
今天用svn提交代码,一直报错! 报错信息: Commit failed (details follow): Error running context: 远程主机强迫关闭了一个现有的连接. 网上的方 ...

flume报错：Space for commit to queue couldn‘t be acquired. Sinks are likely not keeping up with sources

1.问题描述

2.原因

3.解决

flume报错：Space for commit to queue couldn‘t be acquired. Sinks are likely not keeping up with sources相关推荐

最新文章

热门文章