1.Hue 中运行oozie工作流执行spark 报错 local class incompatible

一、问题

在hue中运行oozie工作流执行spark任务时，报错java.io.InvalidClassException: ; local class incompatible: stream classdesc serialVersionUID = -4485890319152472656, local class serialVersionUID = -7506356125639619411

日志：

2021-01-15 15:20:09.104 [task-result-getter-0] WARN  o.a.spark.scheduler.TaskSetManager - Lost task 1.0 in stage 1.0 (TID 11, bigdata001, executor 3): java.io.InvalidClassException: com.qf.ca.cadp.workflows.flight_match.job.FlightSegmentDivideSpark; local class incompatible: stream classdesc serialVersionUID = -4485890319152472656, local class serialVersionUID = -7506356125639619411at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1933)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:490)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)at org.apache.spark.scheduler.Task.run(Task.scala:121)at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1363)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:748)

二、解决

找到hdfs上工作流文件夹，发现在我要执行的workflow的lib目录下存在多个不同版本的jar包，而lib下要求只能有一个jar

这里的路径请根据自己情况修改

hdfs dfs -ls /workflow/lib
app.jar
app-bak1.jar
app-bak2.jar

1 重命名其他版本的jar包

重命名规则，必须在.jar后加字符串，建议拼上时间

这里的路径请根据自己情况修改

hdfs dfs -mv /workflow/lib/app-bak1.jar /workflow/lib/app.jar-bak20201201
hdfs dfs -mv /workflow/lib/app-bak2.jar /workflow/lib/app.jar-bak20201230

2 重跑工作流

重新执行，发现已经可以正常执行了

1.Hue 中运行oozie工作流执行spark 报错 local class incompatible相关推荐

解决docker中运行scrapy使用chrome selenium报错InvalidSessionIdException: Message: invalid session id
本来在本地运行好好的scrapy,结果移植到docker中就报错,完整报错是:selenium.common.exceptions.InvalidSessionIdException: Message ...
在cmd中运行javac编译java文件报错：编码GBK的不可映射字符、非法字符: \65279
操作背景:我在eclipse建立了个HelloWorld.java文件,格式UTF-8,然后复制保存到C:\Users\alex\test目录下,在此处运行按住Shift+右键调出cmd命令窗口,输入 ...
ubuntu20.04_ROS中运行gazebo控制机器人模型报错
1.无法启动类型为[controller_ manager/spawner]的节点:controller_ manager ERROR: cannot launch node of type [con ...
hbase 导入oracle,hue中使用oozie执行sqoop job将oracle中的数据增量导入hbase
在cdh hue中使用oozie执行sqoop job时遇到下面的问题. sqoop命令 job --meta-connect jdbc:hsqldb:hsql://bigdata02:16000/s ...
Spark 中 JVM 内存使用及配置详情、spark报错与调优、Spark内存溢出OOM异常
一般在我们开发spark程序的时候,从代码开发到上线以及后期的维护中,在整个过程中都需要涉及到调优的问题,即一开始需要考虑如何把代码写的更简洁高效调优(即代码优化),待开发测试完成后,提交任务时综合考 ...
HQL语句中数据类型转换，及hibernate中createQuery执行hql报错
一.HQL语句中数据类型转换: 我们需要从数据库中取出序号最大的记录,想到的方法就是使用order by子句进行排序(desc倒序),然后取出第一个对象,可是当初设计数据库时(我们是在原来的数据库的基 ...
mybatis 中case_解决mybatis case when 报错的问题
在mybatis中使用case when进行条件筛选判断时遇到 Failed to process, please exclude the tableName or statementId. 这样的报 ...
Ubuntu执行脚本报错-bash: ./send.py: /usr/bin/python: bad interpreter: Permission denied
在Ubuntu系统终端中,使用命令行如下命令行运行*.py文件: 执行脚本报错 root@ubuntu:/usr/bin/python# chmod a+x send.py root@ubuntu:/ ...
linux运行不存在的命令报错
在Linux中运行一个命令尝试是否存在,突然报错了 Could not find the database of available applications, run update-command- ...