基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —

一、安装PXF3.3.0.0，这里所安装的pxf的包文件都包含在apache-hawq-rpm-2.3.0.0-incubating.tar.gz里面
下面步骤都是以root身份执行
这里注意，pxf插件要用到tomcat服务，必须使用安装包里面的7.0.62，不能安装或升级为 tomcat8，这会造成依赖的catalina.jar包版本不匹配，以致pxf启动！

安装时由于pxf的包都是el6版本的，但是我用的centos7，所以rpm带参数“--nodeps”以忽略RPM命令对依赖包的检测。

cd /opt/gpadmin/hawq_rpm_packagesrpm -ivh  apache-tomcat-7.0.62-el6.noarch.rpm
rpm -ivh --nodeps pxf-service-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-hdfs-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-hive-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-hbase-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-jdbc-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-json-3.3.0.0-1.el6.noarch.rpm
rpm -ivh --nodeps pxf-3.3.0.0-1.el6.noarch.rpm

二、配置PXF

1，由于hadoop为HDP版本，所以使用hdp相关的jar包配置

[root@ep-bd01 ~]cp  /etc/pxf/conf/pxf-privatehdp.classpath  /etc/pxf/conf/pxf-private.classpath[root@ep-bd01 ~]cp  /etc/pxf/conf/pxf-profiles.xml   /etc/pxf/conf/pxf-profiles.default.xml

2，修改pxf目录所有者：

[root@ep-bd01 ~] chown -R pxf:pxf /opt/pxf-3.3.0.0
[root@ep-bd01 ~] chown -R pxf:pxf /tmp/logs

3，建立软连接目录：

[root@ep-bd01 ~] ln -s /etc/pxf-3.3.0.0/conf /opt/pxf-3.3.0.0/conf
[root@ep-bd01 ~] ln -s /usr/lib/pxf  /opt/pxf-3.3.0.0/lib

4，建立一个init过程需要的目录和template文件

[root@ep-bd01 ~] mkdir /opt/pxf-3.3.0.0/conf-templates
[root@ep-bd01 ~] cp /opt/pxf/conf/pxf-privatehdp.classpath /opt/pxf/conf-templates/pxf-private-hdp.classpath.template

5，修改pxf-service文件为pxf，因为init过程需要建立同名的目录，同时需要修改init.d目录中的链接：

[root@ep-bd01 ~] mv /opt/pxf/pxf-service  /opt/pxf/pxf
[root@ep-bd01 ~] unlink /etc/init.d/pxf-service
[root@ep-bd01 ~] ln -s /opt/pxf/pxf   /etc/init.d/pxf-service

6，修改tomcat/conf目录的权限，同时在pxf目录中建立链接

[root@ep-bd01 ~] chmod 755 -R /opt/apache-tomcat/conf/
[root@ep-bd01 ~] ln -s /opt/apache-tomcat /opt/pxf-3.3.0.0/apache-tomcat

7，编辑/etc/pxf/conf/pxf-env.sh，修改 PARENT_SCRIPT_DIR和LD_LIBRARY_PATH的值

[root@ep-bd01 ~] vim /etc/pxf/conf/pxf-env.sh
export PARENT_SCRIPT_DIR=/opt/pxf-3.3.0.0export PXF_HOME=/opt/pxf-3.3.0.0
export LD_LIBRARY_PATH=/usr/hdp/current/hadoop-client/lib/native:${LD_LIBRARY_PATH}

8，修改pxf脚本文件，设置PXF_HOME

[root@ep-bd01 ~] vim /opt/pxf-3.3.0.0/pxf

export PARENT_SCRIPT_DIR=/opt/pxf-3.3.0.0

export PXF_HOME=/opt/pxf-3.3.0.0

9，编辑/etc/pxf/conf/pxf-public.classpath，添加一系列的jar包

[root@ep-bd01 pxf-3.3.0.0]# vim /etc/pxf-3.3.0.0/conf/pxf-public.classpath
/usr/hdp/current/hadoop-client/lib/commons-beanutils-1.9.3.jar
/usr/hdp/current/hadoop-client/lib/commons-cli-1.2.jar
/usr/hdp/current/hadoop-client/lib/commons-codec-1.11.jar
/usr/hdp/current/hadoop-client/lib/commons-collections-3.2.2.jar
/usr/hdp/current/hadoop-client/lib/commons-compress-1.4.1.jar
/usr/hdp/current/hadoop-client/lib/commons-configuration2-2.1.1.jar
/usr/hdp/current/hadoop-client/lib/commons-io-2.5.jar
/usr/hdp/current/hadoop-client/lib/commons-lang-2.6.jar
/usr/hdp/current/hadoop-client/lib/commons-lang3-3.4.jar
/usr/hdp/current/hadoop-client/lib/commons-logging-1.1.3.jar
/usr/hdp/current/hadoop-client/lib/commons-math3-3.1.1.jar
/usr/hdp/current/hadoop-client/lib/commons-net-3.6.jar
/usr/hdp/current/hadoop-client/lib/jersey-core-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-json-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-server-1.19.jar
/usr/hdp/current/hadoop-client/lib/jersey-servlet-1.19.jar
/usr/hdp/current/hadoop-client/lib/jsr311-api-1.1.1.jar
/usr/hdp/current/hadoop-client/lib/woodstox-core-5.0.3.jar
/usr/hdp/current/hadoop-client/lib/stax2-api-3.1.4.jar
/usr/hdp/current/hadoop-client/lib/htrace-core4-4.1.0-incubating.jar
/usr/hdp/current/hadoop-client/lib/re2j-1.1.jar
/usr/hdp/3.0.0.0-1634/hbase/lib/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-rbf.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-nfs.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-native-client.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-httpfs.jar
/usr/hdp/current/hadoop-hdfs-datanode/hadoop-hdfs-client-3.1.0.3.0.0.0-1634.jar
/opt/pxf-3.3.0.0/lib/pxf-service-3.3.0.0.jar
/opt/pxf-3.3.0.0/lib/pxf-api-3.3.0.0.jar
:wq

10，复制pxf-profiles.xml 为pxf-profiles-default.xml并编辑，添加profile配置

[root@ep-bd05 pxf-3.3.0.0]# cp /etc/pxf/conf/pxf-profiles.xml   /etc/pxf/conf/pxf-profiles-default.xml
[root@ep-bd05 pxf-3.3.0.0]# vim /etc/pxf/conf/pxf-profiles-default.xml
<profiles> <profile><name>HdfsTextSimple</name><description>This profile is suitable for using when reading delimited single line records from plain text fileson HDFS</description><plugins><fragmenter>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</fragmenter><accessor>org.apache.hawq.pxf.plugins.hdfs.LineBreakAccessor</accessor><resolver>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</resolver></plugins></profile><profile><name>HdfsTextMulti</name><description>This profile is suitable for using when reading delimited single or multi line records (with quotedlinefeeds) from plain text files on HDFS. It is not splittable (non parallel) and slower than HdfsTextSimple.</description><plugins><fragmenter>org.apache.hawq.pxf.plugins.hdfs.HdfsDataFragmenter</fragmenter><accessor>org.apache.hawq.pxf.plugins.hdfs.QuotedLineBreakAccessor</accessor><resolver>org.apache.hawq.pxf.plugins.hdfs.StringPassResolver</resolver></plugins></profile>
</profiles>

三、初始化pxf，必须使用pxf用户

1,设置pxf的密码

passwd pxf

2，初始化，需要使用用户pxf

[root@ep-bd03 pxf]# source /etc/pxf/conf/pxf-env.sh[root@ep-bd03 pxf]# sudo -u pxf service pxf-service init

Generating /opt/pxf-3.3.0.0/conf/pxf-private.classpath file from /opt/pxf-3.3.0.0/conf-templates/pxf-private-hdp.classpath.template ...

cp /opt/pxf/pxf-service/webapps/pxf/WEB-INF/lib/*.jar  /opt/pxf/lib/

四、启动PXF service

1，启动：

sudo -u pxf service pxf-service start

Checking if tomcat is up and running...
tomcat not responding, re-trying after 1 second (attempt number 1)
Checking if PXF webapp is up and running...
PXF webapp is listening on port 51200

2，测试：

使用pxf插件访问已经事先从oracle导入到HDFS上的数据（使用了sqoop的--compress选项，是gz压缩格式，但是HdfsTextSimple可以直接访问），下面是建立hawq外部表的命令，注意路径中的星号。

drop external table ext.yx_bw;
create external table ext.yx_bw (occur_time date, ......    ) location ('pxf://192.168.58.15:51200/var/data/ext/yx_bw/*?profile=hdfstextsimple') format 'text'(delimiter ',' null '');

**注意** ，此处的主机地址，我直接使用的是主机的地址，如果使用主机名称则hawq访问失败，据我观察应该是没有正确转换，一直没能解决此问题，如果哪位大侠知道请一定不吝赐教，先谢过了！如果地址使用location ('pxf://bd05:51200/var/data/ext/yx_bw/*?profile=hdfstextsimple') ，外部表可以建立，但是访问数据时显示如下错误，且没有详细信息，pxf服务的log也找不到访问失败的记录！

epbd=> select * from ext.yx_bw;
ERROR:  remote component error (0): (libchurl.c:897)

下面是系统中的/etc/hosts文件和/etc/host.conf文件，由于本集群可以访问外网，可以看到nslookup返回了错误地址，但是ping和curl访问都是正确的。

root@ep-bd05 pg_log]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.58.11  ep-bd01 bd01
192.168.58.12  ep-bd02 bd02
192.168.58.13  ep-bd03 bd03
192.168.58.14  ep-bd04 bd04
192.168.58.15  ep-bd05 bd05

[root@ep-bd05 pg_log]# cat /etc/host.conf
multi off

[root@ep-bd05 pg_log]# nslookup bd01
Server:         211.137.160.5
Address:        211.137.160.5#53Non-authoritative answer:
Name:   bd01
Address: 211.137.170.246[root@ep-bd05 pg_log]# ping bd01
PING ep-bd01 (192.168.58.11) 56(84) bytes of data.
64 bytes from ep-bd01 (192.168.58.11): icmp_seq=1 ttl=64 time=0.156 ms
64 bytes from ep-bd01 (192.168.58.11): icmp_seq=2 ttl=64 time=0.160 ms
^C
--- ep-bd01 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.156/0.158/0.160/0.002 ms
[root@ep-bd05 pg_log]# curl http://bd01:51200/pxf/v0
Wrong version v0, supported version is v15[root@ep-bd05 pg_log]#

五、可选操作：

1，修改/opt/pxf/pxf-service/conf/catalina.properties，修改 base.shutdown.port

#base.shutdown.port=-1
base.shutdown.port=8005

2，修改/opt/pxf/pxf-service/conf/tomcat-users.xml,给用户tomcat添加角色manager-gui ，以便可以在浏览器中管理webapps

[root@ep-bd01 ~] vim /opt/pxf/pxf-service/conf/tomcat-users.xml
<role rolename="tomcat"/>
<role rolename="manager-gui"/>
<user username="tomcat" password="tomcat" roles="tomcat,manager-gui"/>
<user username="both" password="tomcat" roles="tomcat,role1"/>
<user username="role1" password="tomcat" roles="role1"/>

基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —— 安装HAWQ插件PXF3.3.0.0...相关推荐

ubuntu16.04安装NIVIDIA显卡驱动，cuda8.0，cuDNN6.0以及基于Anaconda安装Tensorflow-GPU
一.安装NAVIDIA驱动安装之前首先得下载NAVIDIA驱动,其中语言要选择Chinese 下载好以后看一下,应该在Download哪里 1.打开终端,先删除旧的驱动 sudo apt-get p ...
Hadoop2.2.0伪分布式环境搭建（附：64位下编译Hadoop-2.2.0过程）
Hadoop2.2.0伪分布式环境搭建: 写在前面:Hadoop2.2.0默认是支持32位的OS,如果想要在64位OS下运行的话,可以通过在64位OS下面编译Hadoop2.2.0来实现,编译的操作步 ...
基于CentOs下的Hadoop完全分布式集群环境搭建
Hadoop配置一.准备工作 1.下载虚拟机,安装centos 在VM中新建4台虚拟机(内存建议2-4G) VMware14下载提取码:e6dn centos6.8和centos7下载提取码:4 ...
Ubuntu LNMPR环境搭建(Ubuntu 20.04 + Nginx1.8 + PHP 7.3 + Mysql8.0 + Redis5.0)
本文环境 Windows10,Ubuntu 20.04,Nginx1.8,PHP 7.3,Mysql8.0,Redis5.0\ 不懂的可以评论或联系我邮箱:owen@owenzhang.com\ 著作 ...
Ubuntu 12.04安装下载工具 UGet 1.8.0 及 aria2用法
1. aria2用法 aria2是我今天新学到的一个命令行的下载工具,虽然还没研究透,但是他绝对比wget更好. aria2 是 Linux 下一个不错的高速下载工具.由于它具有分段下载引擎,所以支持 ...
Ubuntu 20.04 安装CUDA11.1 和cudnn 8.0.5
Install CUDA cuda 11.1 百度网盘链接密码: bo6f sudo sh cuda_11.1.0_455.23.05_linux.run 把第一个X,用enter键去掉,去掉之后, ...
optee3.12.0 qemu_v8的环境搭建篇(ubuntu20.04)
快速链接: .
ubuntu18.04安装gcc7.3.0g++7.3.0
代码 sudo apt-get update sudo apt-get install aptitude sudo aptitude install gcc sudo apt install gcc ...
基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库01 —— 准备环境，搭建本地仓库，安装ambari...
一.集群软硬件环境准备: 操作系统: centos 7 x86_64.1804 Ambari版本:2.7.0 HDP版本:3.0.0 HAWQ版本:2.3.0 5台PC作为工作站: ep-bd01 ...

基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —— 安装HAWQ插件PXF3.3.0.0...

基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库04 —— 安装HAWQ插件PXF3.3.0.0...相关推荐

最新文章

热门文章