Apache Ranger源码编译及使用

2024-05-16 08:32:59

Ranger源码编译、使用手册

1 Ranger简介

Apache Ranger提供一个集中式安全管理框架,它可以对Hadoop生态的组件如Hive,Hbase进行细粒度的数据访问控制.通过操作Ranger控制台,管理员可以轻松的通过配置策略来控制用户访问HDFS文件夹、HDFS文件、数据库、表、字段权限.这些策略可以为不同的用户和组来设置,同时权限可与hadoop无缝对接.

2 准备

2.1 环境说明

1 Ranger源码编译依赖如下linux组件:maven,git,gcc,mysql

2 安装git和gcc时采用yum安装,请配置好本地源

3 安装mysql是请确保清理好系统自带的mysql.

2.1.1虚拟机里Linux系统版本

[root@localhost ranger-0.5.0-usersync]# cat /etc/issue | grep Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.1.2 JDK版本

[root@localhost native]# java -version

java version "1.7.0_67"

注:官网强调一定是1.7以上版本.

Java(TM) SE RuntimeEnvironment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-BitServer VM (build 24.65-b04, mixed mode)

2.1.3mysql版本

[root@localhost native]# mysql -uroot -proot-e"select version()";

Warning: Using a password onthe command line interface can be insecure.

1 Mysql 驱动为mysql-connector-java-5.1.31-bin.jar

2 改jar被重命名后放置在/usr/share/java/内被其它Ranger插件共享

2.1.4 Maven版本

[root@localhost bin]# mvn -version

Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9;2014-02-15T01:37:52+08:00)

Maven home: /root/maven-3.2.1

Java version: 1.7.0_67,vendor: Oracle Corporation

Java home:/root/jdk1.7.0_67/jre

Default locale: en_US,platform encoding: UTF-8

OS name: "linux",version: "2.6.32-431.el6.x86_64", arch: "amd64", family:"unix"

2.1.5 git版本

[root@localhost native]# git version

git version 1.7.1

2.2 编译准备

2.2.1 安装maven

[root@localhost ~]# cd /root

#https://maven.apache.org/download.cgi 最新版

#http://apache.opencas.org/maven/binaries/apache-maven-3.2.1-bin.tar.gz

tar –zxvf apache-maven-3.2.1-bin.tar.gz

mv apache-maven-3.2.1-bin maven-3.2.1

#修改环境变量,在~/.bash_profile里定义MAVEN_HOME并追加到PATH里

export MAVEN_HOME=/root/maven-3.2.1

:$MAVEN_HOME/bin:$PATH

#source环境变量,测试maven版本

source ~/.bash_profile

mvn –version

2.2.2 安装git

这里通过本地源yum方式安装.

yum install git

注:Linux本地源配置见下:

1) 虚拟机加载Linux ISO镜像

2) 找到rom对应设备名

[root@localhost ~]# lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT

sr0 11:0 1 1024M 0 rom

sr1 11:1 1 3.6G 0 rom

sr2 11:2 1 1024M 0 rom

sda 8:0 0 8G 0 disk

?..sda1 8:1 0 500M 0 part /boot

?..sda2 8:2 0 7.5G 0 part

?..VolGroup-lv_root (dm-0) 253:0 0 6.7G 0 lvm /

?..VolGroup-lv_swap (dm-1) 253:1 0 816M 0 lvm [SWAP]

3) 创建目录

[root@localhost ~]#mkdir –p /mnt/cdrom/

4) 挂载镜像

mount -t iso9660 /dev/sr1 /mnt/cdrom

5) 配置linux更新源,/etc/yum.repos.d,修改成如下:

vi /etc/yum.repos.d/redhat.repo

[rhel-source]

name=Redhat

baseurl=file:///mnt/cdrom/

enabled=1

gpgcheck=1

gpgkey=file:///mnt/cdrom//RPM-GPG-KEY-redhat-release

6) 更新更新源

yum clean all

yum update list

2.2.3 安装gcc

yum install gcc

2.2.4 安装mysql

1) 安装Mysql服务、客户端

rpm –ivh MySQL-shared-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-shared-compat-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-server-5.6.14-1.el6.x86_64.rpm

rpm –ivh MySQL-client-5.6.14-1.el6.x86_64.rpm

2) 启动mysql服务

service mysql start

3) 修改mysql初始密码,先找到安装时的初始密码,在修改成自己的密码

[root@localhost ~]#cat /root/.mysql_secret

# The random password set for the root user at Tue Dec 2221:17:22 2015 (local time):RUmKBqcY

mysql –uroot -p RUmKBqcY

set password=password(‘root’)

3 编译

3.1编译中

1) 拷贝ranger源代码

[root@localhost ~]# cd ~

git clone https://github.com/apache/incubator-ranger.git

cd incubator-ranger

git checkout ranger-0.5

2) 编译ranger源代码

cd ~/incubator-ranger

export MAVEN_OPTS="-Xmx512M"

export JAVA_HOME= /root/jdk1.7.0_67

export PATH=$JAVA_HOME/bin:$PATH

mvn clean compile package assembly:assembly install

ls target/*.tar.gz

[root@localhost ~]#ls/root/incubator-ranger/target/*.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hbase-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kafka-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-kms.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-knox-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-migration-util.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-solr-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-src.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-storm-plugin.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz

/root/incubator-ranger/target/ranger-0.5.0-yarn-plugin.tar.gz

3.2 编译问题

1) 编译过程异常缓慢,一般要3-4天时间

2) 如果出现异常不好定位,可在maven参数里加-X以debug模式诊断

3) 源码编译的相关问题及解决方案

a) Failedto execute goal on project ranger-hdfs-plugin: Could not resolve dependenciesfor project

security_plugins.ranger-hdfs-plugin:

ranger-hdfs-plugin:jar:0.5.0:The following artifacts could not be resolved:

org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde,eigenbase:eigenbase-properties:jar:1.1.4,net.hydromatic:linq4j:jar:0.4,net.hydromatic:quidem:jar:0.1.1:

Could not transfer artifactorg.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde from/to conjars(http://conjars.org/repo): conjars.org:Unknownhost conjars.org -> [Help 1]

解决方案:手动下载错误提示里的jar并拷贝到相应的m2目录内.

源:http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/

目标:/root/.m2/repository/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde

源:http://conjars.org/repo/eigenbase/eigenbase-properties/1.1.4/

目标:/root/.m2/repository/eigenbase/eigenbase-properties/1.1.4/

源:http://conjars.org/repo/net/hydromatic/linq4j/0.4/

目标:/root/.m2/repository/net/hydromatic/linq4j/0.4/

源:http://conjars.org/repo/net/hydromatic/quidem/0.1.1/

目标:/root/.m2/repository/net/hydromatic/quidem/0.1.1/

b)[ERROR]error: error reading /root/.m2/repository/org/json/json/20090211/json-20090211.jar;zip file is empty

[ERROR] -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException:Failed to execute goalorg.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) onproject ranger-hdfs-plugin: Compilation failure

error: error reading/root/.m2/repository/org/json/json/20090211/json-20090211.jar; zip file isempty

atorg.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

解决方案:手动下载错误提示里的json-20090211.jar并拷贝到相应的m2目录内.

c) Runningorg.apache.hadoop.crypto.key.kms.server.TestKeyAuthorizationKeyProvider此步耗时较长,大约20分钟左右

4 配置

Ranger在solr里存储日志,RangerAdmin UI依赖solr组件完成审计日志的查询,所以需要先安装和配置好Solr

注:目前(HDFS-Plugin)的测试日志审计时没选择Solr方式,但还是先配置好Standalone模式的solr.

4.1 Solr或SolrCloud安装配置

cd /root/incubator-ranger/security-admin/contrib/solr_for_audit_setup

#打开install.properties文件,修改参数的值如下所示:

vi install.properties

JAVA_HOME=/root/jdk1.7.0_67

SOLR_INSTALL=true

SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz

SOLR_INSTALL_FOLDER=/opt/solr

SOLR_RANGER_HOME=/opt/solr/ranger_audit_server

SOLR_DEPLOYMENT=standalone

SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data

SOLR_LOG_FOLDER=/var/log/solr/ranger_audits

SOLR_MAX_MEM=2g

#安装单节点的solr

./ setup.sh

#按照如下安装提示启动、停止solr服务

cat/opt/solr/ranger_audit_server/install_notes.txt

/opt/solr/ranger_audit_server/scripts/start_solr.sh

#验证solr服务是否可正常使用

lsof –i:6083

http://192.168.56.101:6083

#确保防火墙已经关闭

chkconfig iptables off

4.2 Ranger Admin安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-admin.tar.gz /root

tar –zxvf ranger-0.5.0-admin.tar.gz

cd ranger-0.5.0-admin

#打开Ranger Admin里install.properties文件,修改参数的值如下所示:

vi install.properties

setup_mode=SeparateDB

DB_FLAVOR=MYSQL

db_root_user=root

db_root_password=root

db_host=localhost

db_name=ranger

db_user=root

db_password=root

audit_store=db

audit_db_name=ranger_audit

audit_db_user=root

audit_db_password=root

policymgr_external_url=http://localhost:6080

policymgr_http_enabled=true

unix_user=ranger

unix_group=ranger

#安装Ranger Admin

#启动Ranger Admin服务

ranger-admin start

#验证Ranger Admin服务,如果出现Ranger的登录界面,说Okay了.注:用户名/密码 admin/admin

lsof –i:6080

http://192.168.56.101:6080

4.3 Ranger-usersync安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-usersync.tar.gz /root/

tar –zxvf ranger-0.5.0-usersync.tar.gz

cd ranger-0.5.0-usersync

#打开usersync Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SYNC_SOURCE=unix

#同步周期,1分钟

SYNC_INTERVAL=1

logdir=/var/log/ranger/usersync

#安装usersync Plugin

#启用usersync Plugin插件

./ranger-usersync-services.sh start

4.4 HDFS-Plugin安装配置

cp/root/incubator-ranger/target/ranger-0.5.0-hdfs-plugin.tar.gz /root

tar –zxvf ranger-0.5.0-hdfs-plugin.tar.gz

cd ranger-0.5.0-hdfs-plugin

#打开HDFS Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://localhost:6080

SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jar

REPOSITORY_NAME=hadoopdev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

#组件对应的用户,这里设置为空.一般Hadoop的内置用户是HDFS或则hadoop

CUSTOM_USER=root

CUSTOM_GROUP=root

#启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]#./enable-hdfs-plugin.sh

Customuser and group are not available, using default user and group.

ERROR:Unable to find the conf directory of component [hadoop]; dir [/root/hadoop/conf] not found.

Exitinginstallation.

注:这里报错,需要额外将HADOOP的conf做个软连接到/root/hadoop/conf.

ln-s /root/hadoop-2.7.1/etc/hadoop/ /root/hadoop/conf

#再次启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

Custom user and group are not available,using default user and group.

ERROR: Unable to find the lib directory ofcomponent [hadoop]; dir [/root/hadoop/lib] not found.

Exiting installation.

#这里需要将HDFS Plugin内的jar和HADOOP包含的HDFS jar都指向/root/hadoop/lib

cp /root/ranger-0.5.0-hdfs-plugin/lib/ranger-hdfs-plugin-impl/*.jar/root/hadoop-2.7.1/share/hadoop/hdfs/lib/

mkdir /root/hadoop/lib

ln -s /root/hadoop-2.7.1/share/hadoop/hdfs/lib//root/hadoop/lib/

#再一次启用HDFS Plugin插件

[root@localhost ranger-0.5.0-hdfs-plugin]# ./enable-hdfs-plugin.sh

#验证HDFS Plugin服务,这时登丽Ranger的管理员界面验证下HDFS plugin是够加载成功,发现并没有.

原因是安装HDFS plugin时install.properties文件里定义的REPOSITORY_NAME(值为hadoopdev)并未通过Ranger Admin在HDFS插件里的服务管理里注册成服务(名hadoopdev).

1 登录Ranger Adming

2 点击HDFS plugin的添加按钮

3 定义服务名为hadoopdev,提交其它信息后保存

#再次验证HDFS plugin插件,则发现已经正常加载

1 如果没有安装和开启Ranger-usersync服务的情况下直接测试HDFS赋权权限是不成功的.

2 Ranger Admin的日志文件见 /root/ranger-0.5.0-admin/ews/logs/xa_portal.log

4.5 Hive-Plugin安装配置

先启动hive的metastore和hiveserver2服务

nohup hive --service metastore-hiveconf hive.root.logger=INFO,console > myout1.file 2>&1 &

nohup hiveserver2 -hiveconfhive.root.logger=INFO,console > myout2.file 2>&1 &

[root@localhost ~]# beeline -u"jdbc:hive2://192.168.56.101:10000" -n root -p test

Connectingto jdbc:hive2://192.168.56.101:10000

Connectedto: Apache Hive (version 1.2.1)

Driver:Hive JDBC (version 1.2.1)

Transactionisolation: TRANSACTION_REPEATABLE_READ

Beelineversion 1.2.1 by Apache Hive

0:jdbc:hive2://192.168.56.101:10000> show databases;

+----------------+--+

|database_name |

+----------------+--+

+----------------+--+

2) Ranger-Admin里注册hive plugin的服务

3) 配置、启用hive plugin

cp/root/incubator-ranger/target/ranger-0.5.0-hive-plugin.tar.gz /root

tar -zxvf ranger-0.5.0-hive-plugin.tar.gz

cd ranger-0.5.0-hive-plugin

#打开Hive Plugin里install.properties文件,修改参数的值如下所示:

vi install.properties

POLICY_MGR_URL=http://192.168.56.101:6080

REPOSITORY_NAME=hivedev

XAAUDIT.DB.IS_ENABLED=true

XAAUDIT.DB.FLAVOUR=MYSQL=MYSQL

XAAUDIT.DB.HOSTNAME=localhost

XAAUDIT.DB.DATABASE_NAME=ranger_audit

XAAUDIT.DB.USER_NAME=root

XAAUDIT.DB.PASSWORD=root

[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the conf directory of component [hive]; dir [/root/hive/conf]not found.

Exitinginstallation.

ln -s /root/apache-hive-0.13.0-bin/conf//root/hive/conf

[root@localhostranger-0.5.0-hive-plugin]# ./enable-hive-plugin.sh

Customuser and group is available, using custom user and group.

ERROR:Unable to find the lib directory of component [hive]; dir [/root/hive/lib] not found.

Exitinginstallation.

cp root/ranger-0.5.0-hive-plugin/lib/ranger-hive-plugin-impl/*.jar/root/apache-hive-1.2.1-bin/lib/

mkdir /root/hive/lib

ln -s/root/apache-hive-1.2.1-bin/lib/ /root/hive/lib/

#将生成的hiveserver2-site.xml拷贝到hive的配置目录下

cp /root/hive/conf/*/root/apache-hive-1.2.1-bin/conf/

#如果hive配置目录里有hiveserver2-site.xml,则需要添加如下内容:

<name>hive.security.authorization.enabled</name>

<value>true</value>

</property>

<name>hive.security.authorization.manager</name>

<value>org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory</value>

</property>

<name>hive.security.authenticator.manager</name>

<value>org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator</value>

</property>

<name>hive.conf.restricted.list</name>

<value>hive.security.authorization.enabled,hive.security.authorization.manager,hive.security.authenticator.manager</value>

</property>

5 使用

5.1 HDFS Plugin验证

1 新增linux用户shenl进行测试

useradd shenl

hadoop fs –mkdir /shenl

2 登录Ranger Admin,新增用户shenl

3 登录Ranger Admin,在hadoopdev里注册新的Policy

4 添加该Policy的权限,指定可以对HDFS里的/shenl有R权限

#切换到shenl用户进行put权限测试,应该报错.

5 编辑Policy追加shenl用户的W权限

#此时shenl用户应该可以拥有/shenl目录的put权限

5.2 Hive Plugin验证

1 hive plugin加载验证

2 定义权限策略

3 beeline里权限验证

6 总结

无

Apache Ranger源码编译及使用相关推荐

Ubuntu 16.04源码编译安装Apache 2.4.25教程
这篇文章主要为大家详细介绍了Ubuntu 16.04源码编译安装Apache 2.4.25,具有一定的参考价值,感兴趣的小伙伴们可以参考一下本文为大家介绍了Ubuntu 16.04源码编译安装Apa ...
Linux学习日记——源码编译Apache
[本文为笔者在学习Linux 下的软件安装时,尝试使用源码安装Apache 的过程,事后进行一个小小的总结,发现错误望指正.] 一.典型的源码编译安装软件的过程包括以下3步: 1) 运行 config ...
源码编译php mysql_linux下apache+mysql+php开发环境纯源代码编译搭建(转)
linux下apache+mysql+php开发环境纯源代码编译搭建(转)[@more@] 记录一下我在fedora core 1下通过源代码编译出来的apache+mysql+php开发环境的全部过 ...
apache的源码包编译
获取apache的源码包 1:apache的优化编译 --with-mpm=prefork 如果是跑Php动态网站的话就编译进去 --with-mpm=worker 跑静态页面的参数 2 ...
linux源码编译安装apache( httpd-2.4.53)处理服务器支持 TLS Client-initiated 重协商攻击(CVE-2011-1473)
linux源码编译安装apache 首先我们需要下载相关的依赖包 apr.apr-util.pcre 下载地址 http://archive.apache.org/dist/apr/apr-1.7.0 ...
LAMP源码编译安装之Apache
LAMP源码编译安装之Apache 一.LAMP的基本架构概述 1.LAMP架构 2.各组件的主要作用如下二.编译安装Apache httpd服务 1.关闭防火墙,将安装Apache所需软件包传到/ ...
编译 Apache Spark 源码报错？那是因为你漏掉了关键操作
前言本文隶属于专栏<1000个问题搞定大数据技术体系>,该专栏为笔者原创,引用请注明来源,不足和错误之处请在评论区帮忙指出,谢谢! 本专栏目录结构和参考文献请见1000个问题搞定大数据技 ...
Splunk安装和配置及源码编译安装SVN
Splunk安装和配置 http://my.oschina.net/tuyang/blog/189159 http://my.oschina.net/longniao/blog/82766 http: ...
php 7.0 编译安装mysql_源码编译安装php7.0.5
源码编译安装php7.0.5 源码包下载地址安装epel扩展yum源 [root@xuegod63 ~]# yum clean all [root@xuegod63 ~]# yum list 注:R ...

最新文章

热门文章