spark读hive没有权限异常Permission denied

sparkSQL读取hive库或表时提示没有权限异常

spark@h1:~$ spark-shell
20/03/08 13:29:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://h1:4040
Spark context available as 'sc' (master = spark://h1:7077, app id = app-20200308132939-0009).
Spark session available as 'spark'.
Welcome to____              __/ __/__  ___ _____/ /___\ \/ _ \/ _ `/ __/  '_//___/ .__/\_,_/_/ /_/\_\   version 2.4.4/_/Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162)
Type in expressions to have them evaluated.
Type :help for more information.scala> spark.sql("use liyuan")。。。。
20/03/08 13:29:50 WARN HiveConf: HiveConf of name hive.tez.input.generate.consistent.splits does not exist
20/03/08 13:29:50 ERROR ObjectStore: Version information found in metastore differs 3.1.0 from expected schema version 1.2.0. Schema verififcation is disabled hive.metastore.schema.verification so setting version.
20/03/08 13:29:50 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 1.2.0, comment = Set by MetaStore spark@192.168.1.11
org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Permission denied;at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:214)at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)at org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:141)at org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:136)at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:55)at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)at org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)at org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:258)at org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)... 49 elided
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Permission denied

问题分析

该库是用hive用户创建的，spark用户下无访问权限
查看hdfs目录得到权限如下

spark@h1:~$ hdfs dfs -ls /user/hive/warehouse/
Found 6 items
drwxr-xr-x   - hive supergroup          0 2020-03-08 13:05 /user/hive/warehouse/liyuan.db

权限里已经说明，用户组权限是可读可执行的，怀疑是spark用户不属于hive用户组，查看用户组管理

bigdata@h1:~$ sudo grep 'hive' /etc/group
bigdata:x:1000:hadoop,hive,hbase,sino,whale,zookeeper,spark
hive:x:1003:

发现hive用户组下没有其他用户

问题解决

添加spark用户到hive用户组

bigdata@h1:~$ sudo usermod -a -G hive spark
bigdata@h1:~$

再次使用spark-shell启动spark-sql查询

scala> spark.sql("select * from tb_evt").collect.foreach(println)
。。。
20/03/08 13:51:48 WARN HiveConf: HiveConf of name hive.tez.input.generate.consistent.splits does not exist
[Stage 0:>                                                          (0 + 0) / 2]20/03/08 13:52:03 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
[190900909,20,20200308]
[232323232,89,20200308]
[217787823,90,20200308]

spark读hive没有权限异常Permission denied相关推荐

nginx的权限问题(Permission denied)解决办法
nginx的权限问题(Permission denied)解决办法一个nginx带多个tomcat集群环境,老是报如下错误: failed (13: Permission denied) while ...
远程连接：hive 报错：Permission denied: user=anonymous, access=EXECUTE
远程连接hive Error: Failed to open new session: java.lang.RuntimeException: java.lang.RuntimeException: ...
【Linux】完美解决 nginx 的权限问题( Permission denied)
问题描述 Linux安装nginx过程中,执行./confiture命令时,报错Permission denied(如图) 解决办法 nginx的目录都是root用户,所以执行命令时提升一下权限即可. ...
hadoop 权限错误 Permission denied: user=root, access=WRITE, inode=/:hdfs:super
关于不能执行Hadoop命令并报权限问题执行错误1.Permission denied: user=root, access=WRITE, inode="/":hdfs:supe ...
macos系统中shell脚本权限不足Permission denied的问题
今天安装IntelliJ IDEA时运行脚本遇到的一些问题,简单总结了一下首先尝试运行脚本时一直报错:Permission denied 如下图: 于是我给sh脚本文件添加权限 chmod +x s ...
ubuntu删除提示权限不够permission denied，以及cannot remove “file“，is a directory.
最近在配置深度学习环境,但是不知怎么的装错了软件版本,而且文件夹还被上锁了,于是尝试了很多方法,提示"permission denied",以及"cannot remov ...
PHP 操作文件没有权限：Permission denied
问题在本地使用PHP脚本操作文件可以成功,等放到服务器上就不行. 查看apache日志: #cd /var/log/httpd #tac error_log 显示: cannot create di ...
linux挂载硬盘无权限（permission denied）
问题描述我在TX2的ubuntu系统里挂载一个硬盘之后,可以看到相应文件夹分配了空间,但是无法对其进行写的操作,并提示权限不够(permission denied) 解决方案: 我们只需要将该文件夹 ...
git pull 报错：权限不够 Permission denied (publickey)
一.背景在git 提交的时候,突然就行不通,一直报错: Permission denied (publickey). fatal: Could not read from remote reposi ...

spark读hive没有权限异常Permission denied

sparkSQL读取hive库或表时提示没有权限异常

问题分析

问题解决

spark读hive没有权限异常Permission denied相关推荐

最新文章

热门文章