Flume1.6.0之Error-protobuf-This is supposed to be overridden by subclasses
1.场景:
Apache Flume: Agent 【http-->memory-->hdfs(CDH4)】 (http发送请求,通过内存,然后写到cdh4的hdfs上)
当前flume agent机器有【CDH4】环境的文件(而没有Apache hadoop环境),
故 JAVA_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
2.错误:
2016-05-21 19:31:27,756 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:234)] Creating hdfs://alish1-dataservice-01.mypna.cn:8022/testwjp/2016-05-21/19/FlumeData.1463830281582.tmp
2016-05-21 19:31:27,791 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)
at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)
3.分析:
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
可能出现不同版本的protobuf包
4.验证:
[root@xxx-01 ~]# find / -name protobuf*.jar
/usr/lib/hadoop-hdfs/protobuf-java-2.4.0a.jar
/usr/share/cmf/lib/cdh4/protobuf-java-2.4.0a.jar
/data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar
/data/01/local/apache-tomcat-7.0.42/webapps/logshedcollector/WEB-INF/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/flume-ng/lib/protobuf-java-2.4.1.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/oozie/libtools/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hbase/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/mahout/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client-0.20/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client-0.20/protobuf-java.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client/protobuf-java.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hcatalog/share/webhcat/svr/lib/protobuf-java-2.4.0a.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar
果然有两个版本:
/data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar
/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/lib/protobuf-java-2.4.0a.jar
5.解决方案
根据 http://caiguangguang.blog.51cto.com/1652935/1592804
返回"This is supposed to be overridden by subclasses"这句话是 高版本protobuf-java-2.5.0.jar才会有的,
那么将高版本jar包移除,重新启动flume,模拟数据http请求,就能写进hdfs了.
mv /data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar /data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar.bak
6.反思,有个担忧
flume1.6.0 -->protobuf-java-2.5.0.jar
cdh4.8.6 -->protobuf-java-2.4.0a.jar
cdh5.4.8 -->protobuf-java-2.5.0.jar
把flume的protobuf-java-2.5.0.jar给移除了,相当于 flume使用了2.4.0包,
担忧1.6.0版本flume的会使用2.5.0版本的函数方法,那么到时就会抛错,故就只能等着看了.
7.建议
假如使用CDH的话,个人建议最好使用配套的版本flume,
http://blog.itpub.net/30089851/viewspace-2092318/ 进cloudera查看对应版本
CDH4.8.6 --》 flume 1.4.0
CDH5.4.8 --》 flume 1.5.0
假如使用Apache Hadoop的话,还是要配套的版本flume.
Flume1.6.0之Error-protobuf-This is supposed to be overridden by subclasses相关推荐
- 【原创】flume-1.3.0安装配置以及flume启动说明
Flume介绍 一 flume基本介绍 Apache flume是一个分布式的.可靠的和可用的用来高效收集.同济和移动大量数据从众多不同sources到一个集中的数据存储库的系统. Flume eve ...
- mysql-4.0.20 use on scounix 5.0.7 error :dynamic linker:..:could not open libgthreads.so help!!!!
周五, 2007/07/06 - 17:28 - zhenghz mysql-4.0.20 use on scounix 5.0.7 error :dynamic linker:..:could no ...
- [2016-04-19 15:46:03 - IceHoloReader1.0] Installation error: INSTALL_FAILED_CONFLICTING_PROVIDER [20
[2016-04-19 15:46:03 - IceHoloReader1.0] Installation error: INSTALL_FAILED_CONFLICTING_PROVIDER [20 ...
- webApi前端ajax调用后端返回{“readyState“:0,“status“:0,“statusText“:“error“}解决方案
webApi前端ajax调用后端返回{"readyState":0,"status":0,"statusText":"error& ...
- kafka 集群启动失败:FATAL [KafkaServer id=0] Fatal error during KafkaServer startup.
在测试 kafka 集群时,关闭 kafka 后,再次启动 kafka 集群时,遇到此问题,具体内容如下: FATAL [KafkaServer id=0] Fatal error during Ka ...
- 01【在线日志分析】之Flume-1.7.0源码编译导入eclipse
[前提]: 1.windows 7 安装maven-3.3.9 其中在conf/setting.xml文件添加 D:\software\apache-maven-3.3.9\repositor ...
- Cannot run program /opt/CI/android-sdk-linux/build-tools/26.0.0/aapt: error=13, Permission denied
执行gradle 构建 java.io.IOException: Cannot run program "/opt/CI/android-sdk-linux/build-tools/26.0 ...
- 解决 : Apache Tomcat/8.0.0-RC1 - Error report ... HTTP Status 404
前些天发现了一个巨牛的人工智能学习网站,通俗易懂,风趣幽默,忍不住分享一下给大家.点击跳转到教程. 1.报错: Apache Tomcat/8.0.0-RC1 - Error report HTTP ...
- Android OpenGL Cannot create GL program: 0 GL error: 1282
Android OpenGL create GL program: 0 & GL error: 1282 快速解决 1. 使用GLSurfaceView的话 请在继承类中合适的地方(一般是构造 ...
- 【caffe】 Check failed: error == cudaSuccess (30 vs. 0) unknown error
解决办法 : 加 sudo 用caffe,不报错,那一定要的时假caffe. ubuntu16.04 ,写了一个.py文件调用resnet.caffemodel 来对图片进行分类. 但是在运行时出错, ...
最新文章
- java 重启系统_java 程序,实现重启功能。求大神帮忙,急
- opencv python 多帧降噪算法_防抖技术 | OpenCV实现视频稳流
- AI算法领域常用的39个术语(下)
- [攻克存储] 掌握SDRAM/DDR的结构与寻址
- c++ c6386 缓冲区 溢出_Office 远程溢出漏洞测试与分析
- 梅森素数:千年不休的探寻之旅
- Luogu2680 [NOIP2015 提高组] 运输计划
- js基础-5-数据类型,作用域,优先级
- RuntimeError: no valid convolution algorithms available in CuDNN
- 树莓派串口/RS485通信
- 任天堂Wii有一些伟大的Wii游戏为Wii控制台你有此致
- Swift里的CAP理论和NWR策略应用
- 数据结构与算法(Leetcode链表篇)
- 做最好的自己——摘自李镇西的《做最好的老师》
- VMware 8安装Mac OS X 10.8 Lion
- 《画解数据结构》三十张彩图,画解二叉搜索树
- 12306泄露数据可视化分析
- 英语:日常用语中的错误表达和正确表达
- 字符识别引擎Calamari总结
- 小程序源码:聊天斗图微信表情包