TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit

异常：

java.lang.Exception: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit
   at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit
   at com.cys.TuiSong.TwoMapper.map(TwoMapper.java:18)
   at com.cys.TuiSong.TwoMapper.map(TwoMapper.java:1)
   at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
   at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
   at java.util.concurrent.FutureTask.run(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

异常分析：

在mapper运行的时候,可以用如下的方法得到对应的filesplit,也就能拿到对应的输入路径,等等信息.

(FileSplit)(reporter.getInputSplit()); 0.19

(FileSplit)(context.getInputSplit());0.20

但是如果是使用

MultipleInputs.addInputPath(job, new Path(path),
SequenceFileInputFormat.class, ProfileMapper.class);

在mapper中再使用上面的那种方式,就会报出一个类型转换错误

Java.lang.ClassCastException: org.apache.Hadoop.mapreduce.lib.input.TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit

而我们需要的filesplit实际上就是TaggedInputSplit中的成员变量inputSplit

然而 TaggedInputSplit这个类在社区版中并不是public的,所以我们并不能直接直接拿到对应的信息了

解决方案：

MultipleInputs.addInputPath(job, new Path(path),
SequenceFileInputFormat.class, ProfileMapper.class);替换成：

FileInputFormat.addInputPath(job,new Path(""));即可

TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit相关推荐

Hadoop源代码分析（包mapreduce.lib.input）
接下来我们按照MapReduce过程中数据流动的顺序,来分解org.apache.hadoop.mapreduce.lib.*的相关内容,并介绍对应的基类的功能.首先是input部分,它实现了MapR ...
#Hive报错 WritableStringObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspect
#Hive报错 FAILED: ClassCastException org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableS ...
hadoop错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster 原创hongxiao2016 最后发布于2019-03-30 21:20:5 ...
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
解决方案: ①确保$HADOOP_HOME/etc/hadoop/mapred-site.xml包含下列设置 <property> <name>yarn.app.mapre ...
IncompatibleClassChangFound interface org.apache.hadoop.mapreduce.JobContext, but class was expected
Sqoop实现oracle数据导入hbase时出错在执行sqoop import --connect jdbc:mysql://192.168.1.107:3306/digta \--usernam ...
执行work count程序报错Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
程序执行过程报错:Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster 根据报错提示找到h ...
记录hadoop3.2.2出现Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster问题
问题: hadoop3.2.2运行集群自带wordcount功能时出现以下错误: [xiaobai@hadoop102 hadoop]$ hadoop jar share/hadoop/mapredu ...
Spark读Hive分区表报错：Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist
前言之前删除Hive分区表的时候,直接用hadoop fs -rm -r 命令删除了hdfs上的分区目录,然后用spark读取hive的这个表的时候,就报错了: Caused by: org.apa ...
Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
我发现大部分网络遇到这个问题的时候,是应为org.apache.hadoop.mapreduce.v2.app.MRAppMaster 类没有在启动的时候被加载到,或者说在使用的时候. 不知道有没有人 ...
hive_Decimal64ColumnVector cannot be cast to org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVect
快速解决: set hive.vectorized.execution.enabled = false

TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit

TaggedInputSplit cannot be cast to org.apache.hadoop.mapreduce.lib.input.FileSplit相关推荐

最新文章

热门文章