hive 集成sentry

2019独角兽企业重金招聘Python工程师标准>>>

环境

apache-hive-2.3.3-bin
apache-sentry-2.1.0-bin
1
2
sentry是目前最新的版本，支持hive的最高版本为2.3.3，hive版本如果高于2.3.3，会出一些版本兼容问题[亲测]

hive快速安装
wget http://mirrors.shu.edu.cn/apache/hive/hive-2.3.3/apache-hive-2.3.3-bin.tar.gz
tar -zxvf apache-hive-2.3.3-bin.tar.gz
1
2
配置hive-site.xml

mv hive-default.xml.template hive-site.xml
mkdir -p /home/xiaobin/soft/apache-hive-2.3.3-bin/tmpdir
vi hive-site.xml

<property>
<name>system:java.io.tmpdir</name>
<value>/home/xiaobin/soft/apache-hive-2.3.3-bin/tmpdir</value>
</property>
<property>
<name>system:user.name</name>
<value>master</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.1.115/hive2?createDatabaseIfNotExist=true&useUnicode=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
下载驱动

yum install mysql-connector-java -y
查看安装路径
[root@cluster-master conf]# rpm -qa|grep mysql-connector-java
mysql-connector-java-5.1.25-3.el7.noarch
[root@cluster-master conf]# rpm -ql mysql-connector-java-5.1.25-3.el7.noarch
/usr/share/doc/mysql-connector-java-5.1.25
/usr/share/doc/mysql-connector-java-5.1.25/CHANGES
/usr/share/doc/mysql-connector-java-5.1.25/COPYING
/usr/share/doc/mysql-connector-java-5.1.25/docs
/usr/share/doc/mysql-connector-java-5.1.25/docs/README.txt
/usr/share/doc/mysql-connector-java-5.1.25/docs/connector-j.html
/usr/share/doc/mysql-connector-java-5.1.25/docs/connector-j.pdf
/usr/share/java/mysql-connector-java.jar
/usr/share/maven-fragments/mysql-connector-java
/usr/share/maven-poms/JPP-mysql-connector-java.pom
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
copy mysql-connector驱动

cp /usr/share/java/mysql-connector-java.jar apache-hive-2.3.3-bin/lib/
1
创建元数据数据库

mysql> create database hive2;
Query OK, 1 row affected (0.01 sec)
1
2
初始化元数据

schematool -dbType mysql -initSchema
1
sentry安装
下载
http://sentry.apache.org/general/downloads.html
wget http://apache.01link.hk/sentry/2.1.0/apache-sentry-2.1.0-bin.tar.gz
tar -zxvf apache-sentry-2.1.0-bin.tar.gz
1
2
3
config
cp sentry-site.xml.service.example sentry-site.xml
vi sentry-site.xml

<property>
<name>sentry.hive.server</name>
<value>server1</value>
</property>

<property>
<name>sentry.verify.schema.version</name>
<value>true</value>
</property>

<property>
<name>sentry.service.allow.connect</name>
<value>hive,impala,hue,hdfs</value>
<description>comma separated list of users - List of users that are allowed to connect to the service (eg Hive, Impala) </description>
</property>

<property>
<name>sentry.store.jdbc.url</name>
<value>jdbc:mysql://localhost:3306/sentry</value>
<description>JDBC connection URL for the backed DB</description>
</property>

<property>
<name>sentry.store.jdbc.user</name>
<value>sentry</value>
<description>The username of the user that connects to the Sentry database</description>
</property>

<property>
<name>sentry.store.jdbc.password</name>
<value>sentry</value>
<description>Sentry password for backend JDBC user </description>
</property>

<property>
<name>sentry.service.server.keytab</name>
<value></value>
<description>Keytab for service principal</description>
</property>

<property>
<name>sentry.service.server.rpcport</name>
<value>8038</value>
<description> TCP port number for service</description>
</property>

<property>
<name>sentry.service.server.rpcaddress</name>
<value>0.0.0.0</value>
<description> TCP interface for service to bind to</description>
</property>

<property>
<name>sentry.store.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
<description>Backend JDBC driver - org.apache.derby.jdbc.EmbeddedDriver (only when dbtype = derby) JDBC Driver class for the backed DB</description>
</property>

<property>
<name>sentry.service.admin.group</name>
<value>hive,impala,hue,hdfs</value>
<description>Comma separates list of groups. List of groups allowed to make policy updates</description>
</property>

<property>
<name>sentry.store.group.mapping</name>
<value>org.apache.sentry.provider.common.HadoopGroupMappingService</value>
<description>
Group mapping class for Sentry service. org.apache.sentry.provider.file.LocalGroupMapping service can be used for local group mapping. </description>
</property>

<property>
<name>sentry.store.group.mapping.resource</name>
<value> </value>
<description> Policy file for group mapping. Policy file path for local group mapping, when sentry.store.group.mapping is set to LocalGroupMapping Service class.</description>
</property>

<property>
<name>sentry.service.security.mode</name>
<value>none</value>
<description>Options: kerberos, none. Authentication mode for Sentry service. Currently supports Kerberos and trusted mode </description>
</property>

<property>
<name>sentry.service.server.principal</name>
<value> </value>
<description>Service Kerberos principal</description>
</property>

<property>
<name>sentry.service.web.enable</name>
<value>true</value>
<description>Enable web service</description>
</property>

<property>
<name>sentry.service.web.authentication.type</name>
<value>NONE</value>
<description>Options: kerberos, NONE. Authentication mode for Sentry web service.</description>
</property>

<property>
<name>sentry.service.web.authentication.kerberos.keytab</name>
<value></value>
<description>Keytab for web service principal</description>
</property>

<property>
<name>sentry.service.web.authentication.kerberos.principal</name>
<value></value>
<description>Web service Kerberos principal</description>
</property>

<property>
<name>sentry.service.web.authentication.allow.connect.users</name>
<value></value>
<description>comma separated list of users - List of users that are allowed to connect to the web service (eg Hive, Impala) </description>
</property>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
创建sentry元数据数据库

Create Database sentry;
Create User sentry Identified By 'sentry';
Grant All On sentry.* To sentry@'localhost' Identified By 'sentry';
Grant All On sentry.* To sentry@'%' Identified By 'sentry';
flush privileges;
1
2
3
4
5
复制mysql-connector驱动

cp mysql-connector-java.jar apache-sentry-2.1.0-bin/lib/
1
初始化元数据

sentry --command schema-tool --conffile apache-sentry-2.1.0-bin/conf/sentry-site.xml --dbType mysql --initSchema

1
2
启动service

./sentry --command service --conffile apache-sentry-2.1.0-bin/conf/sentry-site.xml
1
查看是否启动成功

netstat -anpl|grep 8038
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8038 0.0.0.0:* LISTEN 11950/java
1
2
3
4
hive集成sentry
copy sentry 客户端配置文件

cp apache-sentry-2.1.0-bin/conf/sentry-site.xml.hive-client.example apache-hive-2.3.3-bin/conf/
cd apache-hive-2.3.3-bin/conf/
mv sentry-site.xml.hive-client.example sentry-site.xml
1
2
3
配置$HIVE_HOME/conf/sentry-site.xml

<property>
<name>sentry.service.client.server.rpc-addresses</name>
<value>localhost</value>
<description> TCP address of the sentry store server</description>
</property>

<property>
<name>sentry.service.client.server.rpc-port</name>
<value>8038</value>
<description>Port # of the sentry store server</description>
</property>

<property>
<name>sentry.service.client.server.rpc-connection-timeout</name>
<value>200000</value>
<description>Client timeout default(200000) RPC connection timeout in milisecs</description>
</property>

<property>
<name>sentry.metastore.service.users</name>
<value>hive</value>
<description>
Comma separated list of users
List of service users (eg hive, impala) to bypass
the Sentry metastore authorization. These
services handle the metadata authorization
on their side.
</description>
</property>

<property>
<name>sentry.hive.provider</name>
<value>org.apache.sentry.provider.file.HadoopGroupResourceAuthorizationProvider</value>
<description> Deprecated name: hive.sentry.provider. Group mapping which should be used at client side</description>
</property>

<property>
<name>sentry.hive.server</name>
<value>server1</value>
<description> Deprecated name: hive.sentry.server. Defaut: HS2. Hive Server2 Server identifier like "server1"</description>
</property>

<property>
<name>sentry.hive.failure.hooks</name>
<value> </value>
<description>Deprecated Name: hive.sentry.failure.hooks</description>
</property>

<property>
<name>sentry.hive.testing.mode</name>
<value>true</value>
</property>

<property>
<name>sentry.hive.provider.backend</name>
<value>org.apache.sentry.provider.db.SimpleDBProviderBackend</value>
<description> Options: {org.apache.sentry.provider.db.SimpleDBProviderBackend, org.apache.sentry.provider.file.SimpleFileProviderBackend}
Privilege provider to be used, we support file based or db based
</description>
</property>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
vi hive-site.xml

<property>
<name>hive.metastore.pre.event.listeners</name>
<value>org.apache.sentry.binding.metastore.MetastoreAuthzBinding</value>
</property>

<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener</value>
</property>

<property>
<name>hive.server2.enable.impersonation</name>
<value>true</value>
</property>

<property>
<name>hive.security.authorization.task.factory</name>
<value>org.apache.sentry.binding.hive.SentryHiveAuthorizationTaskFactoryImpl</value>
</property>

<property>
<name>hive.server2.session.hook</name>
<value>org.apache.sentry.binding.hive.HiveAuthzBindingSessionHook</value>
</property>

<property>
<name>hive.sentry.conf.url</name>
<value>file:///home/xiaobin/soft/apache-hive-2.3.3-bin/conf/sentry-site.xml</value>
</property>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
copy sentry jars

cp apache-sentry-2.1.0-bin/lib/sentry-*.jar apache-hive-2.3.3-bin/lib/
cp apache-sentry-2.1.0-bin/lib/shiro-* apache-hive-2.3.3-bin/lib/

1
2
3
启动hiveserver2

hiveserver2 --hiveconf hive.root.logger=INFO,console
1
查看hiveserver2监听端口

netstat -anpl|grep 10000
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN
---------------------

使用hive用户登陆,在这个hive用户是在$HIVE_HOME/conf/sentry-site.xml中配置，

<property>
<name>sentry.metastore.service.users</name>
<value>hive</value>
</property>
1
2
3
4
，hive是用于授权的账号，可以理解为超级用户

beeline -u 'jdbc:hive2://localhost:10000' -n hive
1
查看所有roles，当前没有任何role

0: jdbc:hive2://localhost:10000> show roles;
+-------+
| role |
+-------+
+-------+
1
2
3
4
5
创建admin role:admin_role,

create role admin_role;
GRANT ALL ON SERVER server1 TO ROLE admin_role;
1
2
admin_role拥有server1上的所有权限，server1是再sentry-site.xml中配置

<property>
<name>sentry.hive.server</name>
<value>server1</value>
</property>
1
2
3
4
简单理解拥有admin_role的用户组，拥有所有权限
将hive用户组设置为管理员用户,并使用hive用户创建数据库test

GRANT ROLE admin_role TO GROUP hive;
create database test;

0: jdbc:hive2://localhost:10000> create role xn_role;
No rows affected (0.095 seconds)
0: jdbc:hive2://localhost:10000> GRANT ROLE xn_role TO GROUP xn;
No rows affected (0.118 seconds)
1
2
3
4
xn这个拥有xn_role,但是xn_role没有任何权限

使用xn用户登陆
beeline -u 'jdbc:hive2://localhost:10000' -n xn

show databases没有任何库列表输出
0: jdbc:hive2://localhost:10000> show databases;
+----------------+
| database_name |
+----------------+
| default |
+----------------+
1 row selected (0.71 seconds)

并且也没有建库权限

0: jdbc:hive2://localhost:10000> create database xn;
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User xn does not have privileges for CREATEDATABASE
The required privileges: Server=server1->action=create->grantOption=false; (state=42000,code=40000)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
现在用hive用户账户创建数据库xn,并将xn_role的权限附给xn;

0: jdbc:hive2://localhost:10000> create database xn
. . . . . . . . . . . . . . . .> ;
No rows affected (0.196 seconds)
0: jdbc:hive2://localhost:10000> GRANT ALL ON DATABASE xn TO ROLE xn_role;
No rows affected (0.1 seconds)
0: jdbc:hive2://localhost:10000> GRANT ROLE xn_role TO GROUP xn;
No rows affected (0.135 seconds)

1
2
3
4
5
6
7
8
使用xn用户登陆

beeline -u 'jdbc:hive2://localhost:10000' -n xn
0: jdbc:hive2://localhost:10000> show databases;
+----------------+
| database_name |
+----------------+
| default |
| xn |
+----------------+
2 rows selected (0.651 seconds)
0: jdbc:hive2://localhost:10000>
1
2
3
4
5
6
7
8
9
10
查看当前用户roles

0: jdbc:hive2://localhost:10000> SHOW CURRENT ROLES;
+----------+
| role |
+----------+
| xn_role |
+----------+
1 row selected (0.119 seconds)
1
2
3
4
5
6
7
查看xn_role拥有的权限

1
2
3
4
5
6
7
8
给xn_role添加表sensitive.events查询权限

GRANT SELECT ON table sensitive.events TO ROLE xn_role;

0: jdbc:hive2://localhost:10000> create table xn.events as select * from sensitive.events;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
No rows affected (18.355 seconds)
0: jdbc:hive2://localhost:10000>

1
2
3
4
5
尝试插入数据

0: jdbc:hive2://localhost:10000> insert into sensitive.events select * from xn.events;
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User xn does not have privileges for QUERY
The required privileges: Server=server1->Db=sensitive->Table=events->action=insert->grantOption=false; (state=42000,code=40000)
1
2
3
4
现在用hive账号给xn_role添加对表sensitive.events的所有权限

0: jdbc:hive2://localhost:10000> GRANT ALL ON table sensitive.events TO ROLE xn_role;
No rows affected (0.083 seconds)
1
2
查看xn用户权限

1
2
3
4
5
6
7
8
可以看到xn已经又有了对表sensitive.events的所有权限

0: jdbc:hive2://localhost:10000> insert into sensitive.events select * from xn.events;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
No rows affected (17.397 seconds)
0: jdbc:hive2://localhost:10000> select * from sensitive.events;
+---------------+-----------------+----------------+----------------+
| events.ip | events.country | events.client | events.action |
+---------------+-----------------+----------------+----------------+
| 10.1.2.3 | US | android | createNote |
| 10.200.88.99 | FR | windows | updateNote |
| 10.1.2.3 | US | android | updateNote |
| 10.200.88.77 | FR | ios | createNote |
| 10.1.4.5 | US | windows | updateTag |
| 10.1.2.3 | US | android | createNote |
| 10.200.88.99 | FR | windows | updateNote |
| 10.1.2.3 | US | android | updateNote |
| 10.200.88.77 | FR | ios | createNote |
| 10.1.4.5 | US | windows | updateTag |
| 10.1.2.3 | US | android | createNote |
| 10.200.88.99 | FR | windows | updateNote |
| 10.1.2.3 | US | android | updateNote |
| 10.200.88.77 | FR | ios | createNote |
| 10.1.4.5 | US | windows | updateTag |
+---------------+-----------------+----------------+----------------+
15 rows selected (0.412 seconds)
0: jdbc:hive2://localhost:10000>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
数据已经插入到表sensitive.events

现在来看下怎样收回权限，首先来收回xn_role对表的所有权限

0: jdbc:hive2://localhost:10000> REVOKE ALL ON Table sensitive.events from role xn_role;
No rows affected (0.125 seconds)
1
2
查看xn_role的权限

0: jdbc:hive2://localhost:10000> SHOW GRANT ROLE xn_role;
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
| database | table | partition | column | principal_name | principal_type | privilege | grant_option | grant_time | grantor |
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
| xn | | | | xn_role | ROLE | * | false | 1540965346000 | -- |
+-----------+--------+------------+---------+-----------------+-----------------+------------+---------------+----------------+----------+
0: jdbc:hive2://localhost:10000> select * from sensitive.events;
Error: Error while compiling statement: FAILED: SemanticException No valid privileges
User xn does not have privileges for QUERY
The required privileges: Server=server1->Db=sensitive->Table=events->action=select->grantOption=false; (state=42000,code=40000)
注意
这里账号xn，test，hive均为linux用户，在指定user使用beeline时，linux系统必须要有对应的用户组，否则会报group not exists的错误，或者授权已经成功，但是权限不生效

转载于:https://my.oschina.net/hblt147/blog/2987608

hive 集成sentry相关推荐

Hue、Hive、Sentry、Airflow、Oozie
本篇博文主要讲解的是Hue这个开源软件.通过阅读本博客,可以从宏观的角度了解Hue这个应用. 本文介绍了什么是Hue,Hue的架构,Hue的特点,在Hue上面使用Hive的优点,在Hue上浏览文件和数 ...
配置安全的Impala集群集成Sentry
本文主要记录配置安全的Impala集群集成Sentry的过程.Impala集群上配置了Kerberos认证,并且需要提前配置好Hive与Kerberos和Sentry的集成: 使用yum安装CDH H ...
Hive集成Tez引擎跑任务出现的问题（Java heap space问题）
问题1:Hive集成Tez引擎跑任务出现异常:TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : att ...
Hive集成HBase详解
摘要 Hive提供了与HBase的集成,使得能够在HBase表上使用HQL语句进行查询插入操作以及进行Join和Union等复杂查询应用场景 1. 将ETL操作的数据存入HBase 2. HBas ...
.net core 集成 sentry 进行异常报警
.net core 集成 sentry 进行异常报警 Intro Sentry 是一个实时事件日志记录和汇集的平台.其专注于错误监控以及提取一切事后处理所需信息而不依赖于麻烦的用户反馈.它分为客户端和 ...
hive集成spark和mysql
参考:http://ihoge.cn/2018/HiveSpark.html 一.Hive安装 1.Hive简介 Hive是Facebook开发的构建于Hadoop集群之上的数据仓库应用,可以将结 ...
Hive+LDAP+Sentry
为什么80%的码农都做不了架构师?>>> 介绍 hadoop生态里面常用的安全认证无非两种kerberos和ldap,kerberos的繁琐已经领教过了,ldap还好在生产中也 ...
Hive集成Mysql作为元数据时，提示错误：Specified key was too long; max key length is 767 bytes...
在进行Hive集成Mysql作为元数据过程中.做全然部安装配置工作后.进入到hive模式,运行show databases.运行正常,接着运行show tables:时却报错. 关键错误信息例如以下: ...
2022-09-07 Hive集成HBase
HQL 语法一.Hive集成HBase <property><name>hive.zookeeper.quorum</name><value>hado ...

hive 集成sentry

hive 集成sentry相关推荐

最新文章

热门文章