spark MetaException(message:Version information not found in metastore. )

项目github地址：bitcarmanlee easy-algorithm-interview-and-practice
欢迎大家star，留言，一起学习进步

尝试将spark与hive结合起来。将相关配置完毕以后，启动spark-sql的过程中，出现了以下错误：

16/07/25 17:19:08 WARN metadata.Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166)at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:116)at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientat org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1234)... 14 more
Caused by: java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)... 20 more
Caused by: MetaException(message:Version information not found in metastore. )at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664)at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:606)at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)at com.sun.proxy.$Proxy12.verifySchema(Unknown Source)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)... 25 more

这个问题折磨了本宝宝好久。根据错误信息来看，肯定是hive元数据相关的错误。而hive的元数据是存在mysql里的。到底问题在哪里勒？
网络上找了一大圈，最后定位到了问题：
在hive-site.xml里，有这么一个配置项：

<property><name>hive.metastore.schema.verification</name><value>true</value>
</property>

查了一下这个配置项的意义：
hive.metastore.schema.verification：强制metastore的schema一致性，开启的话会校验在metastore中存储的信息的版本和hive的jar包中的版本一致性，并且关闭自动schema迁移，用户必须手动的升级hive并且迁移schema，关闭的话只会在版本不一致时给出警告，默认是false不开启；

因为此配置项设置成为了true，所以可能存在jar包版本不一致问题。导致无法正常启动！
将value由true改为false以后，顺利启动spark-shell

最后总结一下经验：
越是高层应用，依赖底层与下游的组件就越多，配置项也越多越复杂，出错的概率自然也就大了很多。
遇到错误以后，快速定位问题很重要。一般错误信息的提示都很长，一定注意仔细看最后一个错误信息。因为往往最后一个错误信息是源头，从而导致后续一系列问题。最开始的时候我就是因为没有特别注意到最后一个错误而浪费了一些时间！

spark MetaException(message:Version information not found in metastore. )相关推荐

报错：MetaException(message:Version information not found in metastore. )
报错背景: CDH安装完成hive后启动失败. 报错现象: [main]: Metastore Thrift Server threw an exception... MetaException(me ...
Spark操作sparkSql报错：metastore.ObjectStore: Version information found in metastore differs 2.3.0 from e
Spark操作sparkSql报错:metastore.ObjectStore: Version information found in metastore differs 2.3.0 from e ...
hive.ql.exec.DDLTask. MetaException(message:java.io.IOException: Attempt to start meta tracker faile
Hive 0.13和HBase 0.98.6.1整合出现错误记录下 hive> CREATE TABLE hbase_table_1(key int, value string) > ST ...
spark-shell报错：Version information found in metastore differs 2.3.0 from expected schema version1.2.0
在spark-shell执行如下语句时候 scala> spark.sql("show tables").show 报错: Version information found ...
Error running query: MetaException(message:Got exception: java.net.ConnectException Call From XXXX
问题截图问题描述 Error: Error running query: MetaException(message:Got exception: java.net.ConnectException ...
解决libuuid.so.1 no version information available问题
一.出错现象在执行svn 或者curl命令时, 会出现"/usr/local/lib/libuuid.so.1: no version information available" ...
linux redhat 6.3: /lib/libz.so.1: no version information available
linux gradle android 构建报错日志: :app:mergeResReleaseAwbsAAPT err(Facade for 1318236100): /data/gradleRe ...
FAILED: Error in metadata: MetaException(message:Got exception: java.net.ConnectException
今天在安装配置玩hive后,进入了hive的命令行执行show databases时抛出一下错误: FAILED: Error in metadata: MetaException(message:G ...
【Hive】Caused by: MetaException(message:Hive metastore database is not initialized. Please use schema
启动Hive报错如下: Caused by: MetaException(message:Hive metastore database is not initialized. Please use ...
/usr/lib64/libssl.so.10: no version information available (required by ./mongod)
启动mongodb时,日志提示以下信息: ./mongod: /usr/lib64/libssl.so.10: no version information available (required b ...

spark MetaException(message:Version information not found in metastore. )

spark MetaException(message:Version information not found in metastore. )相关推荐

最新文章

热门文章