大数据测试平台搭建记录(CDH集群)

服务器ip密码
关闭防火墙以centos7为例：

systemctl stop firewalld.service
systemctl disable firewalld.service

关闭selinux

vim /etc/selinux/config SELINUX=disabled

配置主机名及其ip映射关系

vim /etc/hostname
cdh0xvim /etc/hosts

10.66.207.51 cdh01
10.66.207.78 cdh02
10.66.207.95 cdh03
10.66.207.236 cdh04

配置免密码登录（由于网路不通暂时不配置）

ssh-keygen

ssh-copy-id cdh01
ssh-copy-id cdh02
ssh-copy-id cdh03
ssh-copy-id cdh04

上传需要安装的软件上传到 /opt/cdhsoft

安装jdk（Oracle版本）

yum -y install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpmvim /etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$JAVA_HOME/bin:$PATHsource /etc/profile

Ntp集群搭建（由于网路不通暂时不配置）

rpm -qa | grep ntpyum install ntp -y

主机点

 # For more information about this file, see the man pages# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift# Permit time synchronization with our time source, but do not# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery# Permit all access over the loopback interface.  This could# be tightened as well, but to do so would effect some of# the administrative functions.restrict 127.0.0.1
restrict ::1# Hosts on local network are less restricted.restrict 10.66.207.0 mask 255.255.255.0 nomodify notrap# Use public servers from the pool.ntp.org project.# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburstserver 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.orgrestrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noqueryserver  127.127.1.0
fudge   127.127.1.0 stratum 10#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client# Enable public key cryptography.
#cryptoincludefile /etc/ntp/crypto/pw# Key file containing the keys and key identifiers used when operating# with symmetric key cryptography.
keys /etc/ntp/keys# Specify the key identifiers which are trusted.
#trustedkey 4 8 42# Specify the key identifier to use with the ntpdc utility.
#requestkey 8# Specify the key identifier to use with the ntpq utility.
#controlkey 8# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats# Disable the monitoring facility to prevent amplification attacks using ntpdc# monlist command when default restrict does not include the noquery flag. See# CVE-2013-5211 for more details.# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitorntpdate ntp.sjtu.edu.cnsystemctl start ntpd
systemctl enable ntpd

ntp端口

netstat -unlnp | grep ntpd

查看网络中的NTP服务器，显示客户端和每个服务器的关系

ntpq -p

查看node01的ntp服务状态

ntpstat

从节点

 # For more information about this file, see the man pages# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).driftfile /var/lib/ntp/drift# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburstserver 10.66.207.51restrict 10.66.207.51 nomodify notrap noqueryserver 127.127.1.0
fudge 127.127.1.0 stratum 10#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client# Enable public key cryptography.
#cryptoincludefile /etc/ntp/crypto/pw# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys# Specify the key identifiers which are trusted.
#trustedkey 4 8 42# Specify the key identifier to use with the ntpdc utility.
#requestkey 8# Specify the key identifier to use with the ntpq utility.
#controlkey 8# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitorntpdate -u cdh01systemctl start ntpd
systemctl enable ntpd

ntp端口

netstat -unlnp | grep ntpd

查看网络中的NTP服务器，显示客户端和每个服务器的关系

ntpq -p

查看node01的ntp服务状态

ntpstatchkconfig ntpd on

Mysql搭建（密码 123456）

wget http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql-community-server

启动mysql

systemctl start mysqld.service

查看mysql

systemctl status mysqld.service

添加开机自动启动

systemctl enable mysqld.service

查看密码

#grep ‘password’ /var/log/mysqld.log grep 'temporary password' /var/log/mysqld.log

登录mysql

mysql -uroot -p set global validate_password_policy=0;
set global validate_password_length=1;
alter user 'root'@'localhost' identified by '123456';use mysql;
grant all privileges  on *.* to root@'%' identified by "root";
flush privileges;yum -y remove mysql57-community-release.noarch

配置http搭建本地yum源

yum install -y httpd
yum install -y createrepo systemctl start httpd.service
cd /var/www/html
mkdir -p cm6/6.3.1/redhat7/yum/RPMS/x86_64/
mkdir -p cdh6/6.3.2/parcels/

添加 allkeys 于/var/www/html/cm6/6.3.1/

生成repodata文件夹


cd /var/www/html/cm6/6.3.1/redhat7/yum
createrepo .

配置本地yum源

cd /etc/yum.repos.d/
vim cloudera-manager.repo[cloudera-manager]
name=Cloudera Manager
baseurl=http://cdh01/cm6/6.3.1/redhat7/yum
gpgcheck=0
enabled=1

创建用户cloudera-scm（密码 test123456）

useradd cloudera-scm
passwd cloudera-scm
test123456

免秘钥登录

echo “cloudera-scm ALL=\(root\)NOPASSWD:ALL” >> /etc/sudoers
su cloudera-scm
Exit

安装cm

yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

新建数据库所有的服务不管用到不用到

# scm
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123456';# amon
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon123456';# rman
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman123456';# hue
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123456';# hive
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive123456';# sentry
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry123456';# nav
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav123456';# navms
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms123456';# oozie
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123456';# flush
FLUSH PRIVILEGES;jdbc驱动
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h 10.66.207.51 --scm-host 10.66.207.51 scm scm

配置parcel

cd /opt/cloudera/parcel-repo

上传parcel 更改名字

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

更改文件所有者

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

启动cm

systemctl start cloudera-scm-servertail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

登录cm安装集群

Hue登录新建账号

Krb5安装

主节点

yum install -y krb5-server krb5-workstation krb5-libs

从节点

yum install -y krb5-workstation krb5-libs

配置服务器

vim /var/kerberos/krb5kdc/kdc.conf[kdcdefaults]kdc_ports = 88kdc_tcp_ports = 88[realms]HADOOP.COM = {#master_key_type = aes256-ctsmax_life = 1dmax_renewable_life = 7dacl_file = /var/kerberos/krb5kdc/kadm5.acldict_file = /usr/share/dict/wordsadmin_keytab = /var/kerberos/krb5kdc/kadm5.keytabsupported_enctypes = aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal}

客户端配置

vim /etc/krb5.conf# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/[logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log[libdefaults]dns_lookup_realm = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = truerdns = falsepkinit_anchors = FILE:/etc/pki/tls/certs/ca-bundle.crtdefault_realm = HADOOP.COMudp_preference_limit = 1default_ccache_name = KEYRING:persistent:%{uid}[realms]HADOOP.COM = {kdc = cdh01admin_server = cdh01}[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM

生成数据库

kdb5_util create -s

密码 admin

vim /var/kerberos/krb5kdc/kadm5.acl
*/admin@HADOOP.COM      *[root@cdh01 krb5kdc]# systemctl start krb5kdc
[root@cdh01 krb5kdc]# systemctl start kadmin
[root@cdh01 krb5kdc]# systemctl enable krb5kdc
Created symlink from /etc/systemd/system/multi-user.target.wants/krb5kdc.service to /usr/lib/systemd/system/krb5kdc.service.
[root@cdh01 krb5kdc]# systemctl is-enabled krb5kdc
enabled
[root@cdh01 krb5kdc]# systemctl enable kadmin
Created symlink from /etc/systemd/system/multi-user.target.wants/kadmin.service to /usr/lib/systemd/system/kadmin.service.
[root@cdh01 krb5kdc]# systemctl is-enabled kadmin
enabled

创建用户

kadmin.local -q "addprinc admin/admin"

验证用户

kinit admin/admin

检查用户 klist

创建用户 kadmin.local -q “addprinc clouder-scm/admin”

clouder-scm

集成cdh（页面集成）

查看添加的主体

kadmin.local -q "list_principals"

Sentry配置（权限验证）

添加linux用户

[root@cdh01 krb5kdc]# useradd test01
[root@cdh01 krb5kdc]# useradd test02

权限测试：

大数据测试平台搭建记录(CDH集群)相关推荐

CDH大数据平台搭建之SPARK集群搭建
CDH大数据平台搭建之SPARK集群搭建一.安装规划二.下载三.安装及配置 1.解压 2.修改配置文件四.启动五.JPS检查节点一.安装规划参考CHD大数据平台搭建之集群规划二.下载 ...
CDH大数据平台搭建之KAFKA集群搭建
CDH大数据平台搭建之KAFKA集群搭建一.安装规划二.下载KAFKA 三.安装及配置 1.先安装zookeeper 2.解压 3.新建文件夹 4.修改config目录server.propert ...
ClickHouse数据库培训实战（PB级大数据分析平台、大规模分布式集群架构）
一.ClickHouse数据库培训实战课程 (PB级大数据分析平台.大规模分布式集群架构)视频教程为满足想学习和掌握ClickHouse大数据分析专用的数据库,风哥特别设计的一套比较系统的Click ...
大数据平台搭建，cdh选型配置
天天干后端,又干前端,本行大数据,差点生疏了,哦吼~正当我发愁之际. 领导安排任务了,下个月去某控上市公司.他们需要从0 开始做大数据平台,主要工作内容如下, 平台搭建,数据迁移,数据分析与挖掘.然后 ...
CDH大数据平台搭建之集群规划
CDH大数据平台搭建之集群规划前言一.集群规模二.集群规划总结前言话说无规矩不成方圆,搭建CDH大数据平台之前需要的工作很多,首先,你需要计算公司每日的数据量,来确定需要多少服务器,确定好 ...
大数据平台搭建及集群规划
CDH大数据平台搭建之集群规划_码上_成功的博客-CSDN博客_cdh集群规划大数据Hadoop分布式集群部署(详细版)_arnoldmp的博客-CSDN博客_分布式集群部署大数据平台的硬件规划. ...
VMware 搭建大数据测试平台（CDH6.2.1）
文章目录 VMware 搭建大数据测试平台 1.安装虚拟机 1.1 创建一台虚拟机 1.2 开机安装CentOS 1.3 配置NAT网络 1.4 配置CentOS系统 1.4.1 修改映射关系 1.4 ...
cdh集群搭建(6.3)
如果你的虚拟机给的内存少于14G,不用往下看了,机器压力非常大,建议先装内存条. 我们做大数据的,机器就是要大硬盘,大内存,高核cpu,其他无所谓,如果要做机器学习,那你显卡还得很好. cdh的好处, ...
搭建ELK日志分析平台（上）—— ELK介绍及搭建 Elasticsearch 分布式集群
笔记内容:搭建ELK日志分析平台(上)-- ELK介绍及搭建 Elasticsearch 分布式集群笔记日期:2018-03-02 27.1 ELK介绍 27.2 ELK安装准备工作 27.3 安装 ...

大数据测试平台搭建记录(CDH集群)

大数据测试平台搭建记录(CDH集群)

关闭selinux

配置主机名及其ip映射关系

Mysql搭建（密码 123456）

启动mysql

查看mysql

添加开机自动启动

查看密码

登录mysql

配置http搭建本地yum源

添加 allkeys 于/var/www/html/cm6/6.3.1/

生成repodata文件夹

配置本地yum源

创建用户cloudera-scm（密码 test123456）

免秘钥登录

安装cm

新建数据库 所有的服务 不管用到不用到

配置parcel

上传parcel 更改名字

更改文件所有者

启动cm

登录cm安装集群

Hue登录新建账号

Krb5安装

主节点

从节点

配置服务器

客户端配置

生成数据库

密码 admin

创建用户

验证用户

检查用户 klist

创建用户 kadmin.local -q “addprinc clouder-scm/admin”

集成cdh（页面集成）

查看添加的 主体

Sentry配置（权限验证）

添加linux用户

大数据测试平台搭建记录(CDH集群)相关推荐

最新文章

热门文章

新建数据库所有的服务不管用到不用到

查看添加的主体