Atlas配置Hive Hook

配置hive-site.xml

在Hive客户端$HIVE_HOME/conf/hive-site.xml中加入内容：

<property><name>hive.exec.post.hooks</name><value>org.apache.atlas.hive.hook.HiveHook</value>
</property>

解压hive-hook包

解压hook包(这里以hive-hook为例)，并拷贝内容到atlas安装目录下
```
tar -zxvf apache-atlas-2.2.0-hive-hook.tar.gz -C /opt/atlas/atlas
```

修改Hive-env.sh

[root@dc6-80-283 conf]# cd /opt/hive/hive/conf
[root@dc6-80-283 conf]# cp hive-env.sh.template hive-env.sh
[root@dc6-80-283 conf]# vim hive-env.sh
# 加入内容
export HIVE_AUX_JARS_PATH=/opt/atlas/atlas/hook/hive/

然后重启hive

复制ATLAS_HOME/conf/atlas-application.properties到HIVE_HOME/conf目录下

cp /opt/atlas/atlas/conf/atlas-application.properties /opt/hive/hive/conf/

执行同步Hive元数据脚本

# 这里同步的是Hive中已有数据的元数据，可以通过此脚本同步过来
[root@dc6-80-283 ]# cd /opt/atlas/atlas/hook-bin/
# 执行脚本导入元数据，期间需要输入atlas的用户名和密码：admin/admin
[root@dc6-80-283 ]# import-hive.sh

2022-06-29T19:30:02,605 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2022-06-29T19:30:02,612 WARN [main] com.zaxxer.hikari.util.DriverDataSource - Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-06-29T19:30:03,057 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
2022-06-29T19:30:03,090 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Starting...
2022-06-29T19:30:03,091 WARN [main] com.zaxxer.hikari.util.DriverDataSource - Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-06-29T19:30:03,118 INFO [main] com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Start completed.
2022-06-29T19:30:03,212 INFO [main] org.apache.hadoop.hive.metastore.ObjectStore - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2022-06-29T19:30:03,336 INFO [main] org.apache.hadoop.hive.metastore.MetaStoreDirectSql - Using direct SQL, underlying DB is MYSQL
2022-06-29T19:30:03,339 INFO [main] org.apache.hadoop.hive.metastore.ObjectStore - Initialized ObjectStore
2022-06-29T19:30:03,519 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,519 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:03,520 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,637 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,637 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:04,638 WARN [main] DataNucleus.MetaData - Metadata has jdbc-type of null yet this is not valid. Ignored
2022-06-29T19:30:06,176 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - Added admin role in metastore
2022-06-29T19:30:06,180 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - Added public role in metastore
2022-06-29T19:30:06,201 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - No user is added in admin role, since config is empty
2022-06-29T19:30:06,347 INFO [main] org.apache.hadoop.hive.metastore.RetryingMetaStoreClient - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=root (auth:SIMPLE) retries=1 delay=1 lifetime=0
2022-06-29T19:30:06,374 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_all_functions
2022-06-29T19:30:06,376 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root    ip=unknown-ip-addr cmd=get_all_functions
2022-06-29T19:30:06,406 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Importing Hive metadata
2022-06-29T19:30:06,406 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_databases: @hive#
2022-06-29T19:30:06,407 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root    ip=unknown-ip-addr cmd=get_databases: @hive#
2022-06-29T19:30:06,413 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 databases
2022-06-29T19:30:06,413 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_database: @hive#default
2022-06-29T19:30:06,413 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root    ip=unknown-ip-addr cmd=get_database: @hive#default
2022-06-29T19:30:06,459 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,717 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Database default is already registered - id=07cf307a-613f-42d2-bf33-12fd26e3bd67. Updating it.
2022-06-29T19:30:06,784 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,793 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_db entity: name=default@primary, guid=07cf307a-613f-42d2-bf33-12fd26e3bd67
2022-06-29T19:30:06,809 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_tables: db=@hive#default pat=.*
2022-06-29T19:30:06,809 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root    ip=unknown-ip-addr cmd=get_tables: db=@hive#default pat=.*
2022-06-29T19:30:06,813 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 tables to import in database default
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_table : tbl=hive.default.t_order
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore.audit - ugi=root    ip=unknown-ip-addr cmd=get_table : tbl=hive.default.t_order
2022-06-29T19:30:06,944 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:06,945 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Table default.t_order is already registered with id 63fb7492-0c8d-4185-b556-8a7a3c6b8f5c. Updating entity.
2022-06-29T19:30:07,052 INFO [main] org.apache.atlas.AtlasBaseClient - method=POST path=api/atlas/v2/entity/ contentType=application/json; charset=UTF-8 accept=application/json status=200
2022-06-29T19:30:07,053 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Updated hive_table entity: name=default.t_order@primary, guid=63fb7492-0c8d-4185-b556-8a7a3c6b8f5c
2022-06-29T19:30:07,053 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Successfully imported 1 tables from database default
Hive Meta Data imported successfully!!!
[root@dc6-80-283 hook-bin]#

可以看到成功导入了数据表t_order

2022-06-29T19:30:06,813 INFO [main] org.apache.atlas.hive.bridge.HiveMetaStoreBridge - Found 1 tables to import in database default
2022-06-29T19:30:06,815 INFO [main] org.apache.hadoop.hive.metastore.HiveMetaStore - 0: get_table : tbl=hive.default.t_order

查看Atlas Web页面

Atlas配置Hive Hook相关推荐

cdh的hive配置文件位置_CDH6配置 Atlas，及 Hive Hook
因为 CDH 社区版不能使用 Navigator,所以需要自己集成一个Apache Atlas. 版本说明 20200818 Updated: 目前最新版是2.1.0,如果 Hive 版本是3.1一下 ...
自定义 Hive Hook
前言在关联hive SQL时,yarn application有多个与其对应,所以需要划分二者对应关系.通过开启hive seesion,解析hive session日志,来解析 hive sql ...
Superset配置hive数据源
1.在uri中配置 hive://localhost:10000/default 2.查询 3.如果你的hive集群是带有kerberos认证的,hive数据源需要这样配置 hive://xxx:xx ...
ubuntu14.04使用MySQL数据库安装配置Hive 1.2.1
ubuntu14.04使用MySQL数据库安装配置Hive 1.2.1 一.准备Mysql已经安装好.本人装的是navicat,图形化界面. 并在本地用root用户建立了test数据库二 ...
hive metastore mysql_Hive初步使用、安装MySQL 、Hive配置MetaStore、配置Hive日志《二》...
一.Hive的简单使用基本的命令和MySQL的命令差不多首先在 /opt/datas 下创建数据 students.txt 1001 zhangsan 1002 lisi 1003 wangwu ...
atlas mysql_使用Atlas配置MySQL读写分离
Atlas是360团队弄出来的一套基于MySQL-Proxy基础之上的代理,修改了MySQL-Proxy的一些BUG,并且优化了很多东西.而且安装方便.配置与注释写的都很详细,并且是都是中文,不过从A ...
mysql 中间件 atlas_mysql中间件atlas配置使用
MySQL所在机器: 192.168.16.70(Master) 192.168.16.74(Slave) 192.168.16.72(atlas) 注意:主从复制需要自行配置 atlas配置使用: ...
Apache Hive 2.1.1 安装配置超详细过程，配置hive、beeline、hwi、HCatalog、WebHCat等组件...
2019独角兽企业重金招聘Python工程师标准>>> 在Docker环境成功搭建了Apache Hadoop 2.8 分布式集群,并实现了NameNode HA.ResourceM ...
优麒麟mysql_Ubuntu kylin优麒麟下配置Hive环境
Ubuntu kylin优麒麟下配置Hive环境 Hive简介什么是Hive hive是基于Hadoop的一个数据仓储工具,可以将结构化的数据文件映射为一张数据表,并提供SQL查询功能,可以将SQL ...

Atlas配置Hive Hook

Atlas配置Hive Hook

Atlas配置Hive Hook相关推荐

最新文章

热门文章