Nutch2.x+Hadoop 2.5.2+Hbase0.94.26(续2)

1.执行bin/nutch generate -topN 5 -crawlId tieba的时候，出现以下错误

java.lang.Exception: java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to org.apache.gora.persistency.Persistent

at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)

Caused by: java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to org.apache.gora.persistency.Persistent

at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(PersistentDeserializer.java:71)

at org.apache.gora.mapreduce.PersistentDeserializer.deserialize(PersistentDeserializer.java:35)

at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:146)

at org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121)

at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:302)

at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170)

at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)

at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

初步怀疑是avrò的版本不匹配，把avrò从1.7.7降级到1.7.6问题依然存在。然后发现执行nutch的时候，classpath里面调用的都是hadoop 2.5.2的jar，而在hadoop-2.5.2/share/hadoop/common/lib/下，avro的版本是1.7.4，把1.7.7版本替换进去，问题解决

2.执行bin/nutch fetch 1421804965-1372033824 -crawlId tieba -threads 50，其中1421804965-1372033824为在hbase shell中执行 get 'tieba_webpage','com.baidu.tieba:http/' 所得f:bid timestamp=1421804970851, value=1421804965-1372033824

此时报错，No agents listed in 'http.agent.name' property

修改nutch-default.properties中的 <name>http.agent.name</name>部分，添加任意字符串

转载于:https://www.cnblogs.com/mactech/p/4239163.html

Nutch2.x+Hadoop 2.5.2+Hbase0.94.26(续2)相关推荐

搭建hbase-0.94.26集群环境
先安装hadoop1.2.1,见http://blog.csdn.net/jediael_lu/article/details/38926477 1.配置hbase-site.xml <prop ...
hadoop1.2.1+zookeeper3.4.6+hbase0.94集群环境搭建
hadoop集群环境搭建参考前面的hadoop集群搭建文档,搭建了三个服务器的集群如下 feixu-master namenode, secondaryNamenode, jobtracke ...
hadoop hive hbase 入门学习 (二)
hadoop 自学系列 hadoop hive hbase 入门学习 (一) hadoop安装.hdfs学习及mapreduce学习 hadoop 软件下载 (hadoo ...
nutch,hbase,zookeeper兼容性问题
nutch-2.1使用gora-0.2.1, gora-0.2.1使用hbase-0.90.4,hbase-0.90.4和hadoop-1.1.1不兼容,hbase-0.94.4和gora-0.2.1 ...
apache-hadoop-1.2.1、hbase、hive、mahout、nutch、solr安装教程
1 软件环境: VMware8.0 Ubuntu-12.10-desktop-i386 jdk-7u40-linux-i586.tar.gz hadoop-1.2.1.tar.gz eclipse-d ...
ubuntu下hbase的伪分布式安装与配置
http://www.cnblogs.com/linjiqin/archive/2013/03/06/2946212.html 1.hadoop版本:hadoop-1.1.1,HBase版本:hbas ...
irms模拟数据生成及数据分析
一.数据准备 1.每天生成随机一个文本,每小时向文本中追加2次数据,每次10万条随机数据生成: 2,32 * * * * bash /mnt/jediael/irms/signalGenerat ...
【Nutch2.3基础教程】集成Nutch/Hadoop/Hbase/Solr构建搜索引擎：安装及运行【集群环境】
1.下载相关软件,并解压版本号如下: (1)apache-nutch-2.3 (2) hadoop-1.2.1 (3)hbase-0.92.1 (4)solr-4.9.0 并解压至/opt/jedi ...
在Eclipse中运行Nutch2.3
参考http://wiki.apache.org/nutch/RunNutchInEclipse 一.环境准备 1.下载nutch2.3源代码 wget http://mirror.bit.edu.c ...

Nutch2.x+Hadoop 2.5.2+Hbase0.94.26(续2)

Nutch2.x+Hadoop 2.5.2+Hbase0.94.26(续2)相关推荐

最新文章

热门文章