cdh6.3安装以及整合spark2、flink1.9

总结：

配置cm-server的yum本地安装路径，使用yum安装好cm-server即可，cm-agent无需手动到节点去安装，cm会自动在节点上安装。
parcels文件不用安装yum本地源，后续直接拷贝到cm server 的/opt/cloudera/parcel-repo/下即可。
jdk手动安装，目录必须是：/usr/java/jdk-version，否则报错
yum需要安装前期的依赖，否则后面agent无法安装，出现莫名其妙的问题，安装依赖后就成功了！

服务器设置

说明：服务器centos7
java version “1.8.0_251”

1.安装依赖

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y

关闭防火墙

sudo systemctl disable firewalld
sudo systemctl stop firewalld

2.配置系统环境

以下需要在每个节点上执行

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'vm.swappiness=10'>> /etc/sysctl.conf   #  这个也要重启# 关闭Security-Enhanced Linux (SELinux)
# 查询是否关闭
getenforce
# 修改/etc/selinux/config
SELINUX=disabled
# 重启

3.开启NTP服务

yum install ntp

编辑/etc/ntp.conf

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

开启服务

sudo systemctl start ntpd
sudo systemctl enable ntpd
# 其他机器可同步dc5同步时间
ntpdate -u <ntp_server>
# 写入系统时间
hwclock --systohc

4.CDH端口情况一览
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_ports.html

5.安装jdk

jdk必须是64bit，注意jdk目录必须是/usr/java/jdkxxxx ，否则安装server将失败

6.安装数据库

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
systemctl start mysqld
systemctl enable mysqld# 设置密码
/usr/bin/mysql_secure_installation

7.安装数据库驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gzwget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar
# 拷贝到目录，如果目录不存在，则新建，并且改名为mysql-connector-java.jar
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

8.执行数据库脚本

执行脚本：

# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>/opt/cloudera/cm/schema/scm_prepare_database.sh -uroot -proot  mysql scm scm scm123# 出现All done, your SCM database is configured correctly!标识成功

创建其他数据库（以后安装服务的时候还是得创建）

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;-- 注意密码最低要求4位以上
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123';

开始安装

离线安装
CDH提供2种方式安装：parcel repositories and package repositories.

安装cloudera manager 要使用yum来安装，6以前可以使用tgz安装，6以后没有tgz包安装了

本地yum安装

本地yum安装cloudera manager

1.安装httpd服务

sudo yum install httpd
sudo systemctl start httpd

2.下载文件

sudo mkdir -p /var/www/html/cloudera-repos/cm6
# 注意cm目前6版本只有6.3.1，cm服务文件可以是6.3.2
# 注意目录是除host外，其他保持一致
sudo mkdir -p /var/www/html/cloudera-repos/将6个文件全部下载到/var/www/html/cloudera-repos/ 这个目录即可(不用创建和下载一样的路径)
下载地址：https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
同时下载https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/ 路径下的cloudera-manager.repo和RPM-GPG-KEY-cloudera  chmod -R ugo+rX /var/www/html/cloudera-repos/
# 在当前目录下执行
createrepo . # 生成yum元数据进行yum安装即可

3.配置本地源
vim cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager 6.3.2
baseurl=http://dc5/cloudera-repos/yum   # 对应到上面rpm本地目录
gpgcheck=0
enabled=1

并将yum配置分发到每个agent节点。
执行yum clean all & yum makecache
yum repolist 查看是否生效

4.开始安装cloddera manager package

yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

5.在server之外节点安装agent

yum install cloudera-manager-agent cloudera-manager-daemons

6.启动server和agent

systemctl start cloudera-scm-server
systemctl stop  cloudera-scm-serversystemctl start cloudera-scm-agent
systemctl stop cloudera-scm-agent

查看日志

 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.logtail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

当出现如下执行ddl脚本，表示启动server成功

2020-05-08 14:20:34,367 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,368 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,376 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,377 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 0
2020-05-08 14:20:40,148 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00001_cmf_schema.mysql.ddl
2020-05-08 14:20:40,160 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 1
2020-05-08 14:20:40,458 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00002_cmf_schema.mysql.ddl
2020-05-08 14:20:40,470 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 2
2020-05-08 14:20:40,628 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00003_cmf_schema.mysql.ddl
2020-05-08 14:20:40,639 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 3
2020-05-08 14:20:40,805 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00004_cmf_schema.mysql.ddl

8.访问admin界面：

http://dc5:7180 输入密码：admin/admin

本地离线parcel

这个步骤可以直接省略，把https://archive.cloudera.com/cdh6/6.3.2/parcels/下载的4个文档放到server中的/opt/cloudera/parcel-repo的目录下即可，如果无效，然后再使用httpd来添加到自定义存储库

使用这种方式前提是已经安装好cloudera manager

1.配置本地parcel repositories

sudo yum install httpd

修/etc/httpd/conf/httpd.conf 在<IfModule mime_module>增加：

AddType application/x-gzip .gz .tgz .parcel

启动httpd服务

sudo systemctl start httpd

2.下载parcel包

下载地址： https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_download.html#cdh_633-download

目前新版本6.3.3需要用户名密码下载，这里用6.3.2版本

总共下载4个文件，放入httpd服务器的目录，这样就能用使用本地httpd提供服务来分发parcel包

# cdh文件
mkdir -p /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/wegt https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel# 这个文件下载后要把后面的1给去掉，否则会从网上下载，而不是使用本地
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 -P CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.shawget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256wget https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json# gplextras文件
mkdir -p /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha1
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha256
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/manifest.json

递归下载文件夹，忽略父文件夹命名参考：
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/version/parcels/ -P /var/www/html/cloudera-repos

修改权限

sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

其他组件的parcel包根据需要下载，这里下载spark2

sudo mkdir -p /var/www/html/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/spark2/parcels/2.3.0.cloudera3/ -P /var/www/html/cloudera-repos
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/spark2

访问地址看看下载的文件是否都有：http://<Web_server>/cloudera-repos/

4.配置Cloudera Manager以使用本地parcels

使用这种方式也得先把cloudera manager安装好

在cdh管理页面打开导航栏中：
Administration > Settings
Category > Parcels

输入本地的地址：http://dc5/cloudera-parcels/cdh6/6.3.2/

输入变更原因，点击保存即可。

配置机器

登录cmf管理页面进行安装

注意，这里agent，可以直接游cm-server来安装。

安装spark2

注意：默认6.3.2 的spark已经是2.4.0版本了，所以已经不用再安装了

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2.html

下载文件到/opt/cloudera/csd

wget http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.4.0.cloudera2.jarwget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/manifest.jsonmv SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1 SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha

重启server服务

systemctl restart cloudera-scm-server

主机-parcel-分配-激活
添加服务即可。

整合Flink

2020年5月7日 15:53:58截至目前官网支持flink1.9.1版本，但是下载要登陆，而是支持cdh7.0.3
文档见：https://docs.cloudera.com/csa/1.1.0/support-matrix/topics/csa-system-req.html

最新版本的需要用户名密码。

这里使用FLINK-1.9.0-csa1.0.0.0-cdh6.3.0版本，具体步骤：

参考：https://cloud.tencent.com/developer/article/1536518

下载地址：https://archive.cloudera.com/csa/

mkdir -p /var/www/html/flink1.9
cd /var/www/html/flink1.9
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/manifest.json
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel.sha# 进入csd，下载jar包
cd /opt/cloudera/csd
wget https://archive.cloudera.com/csa/1.0.0.0/csd/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0.jar

重启服务：systemctl restart cloudera-scm-server
主机-parcel -配置-远程parcel存储库url添加一个：http://dc5/flink1.9/ 保存