【Hadoop HA】搭建Hadoop HA的详细教程
搭建Hadoop HA的详细教程
- 前置工作
- 配置host文件
- ssh免密登录
- JDK的配置
- ZooKeeper配置
- Hadoop HA配置文件
- core-site.xml
- hdfs-site.xml
- mapred-site.xml
- yarn-site.xml
- 启动与测试
前置工作
配置host文件
在每个节点的/etc/hosts
文件下加入
192.168.3.73 master
192.168.3.66 slave1
192.168.3.61 slave2
确保可以相互ping
通
[root@host-192-168-3-73 ~]# ping slave1
PING slave1 (192.168.3.66) 56(84) bytes of data.
64 bytes from slave1 (192.168.3.66): icmp_seq=1 ttl=64 time=0.589 ms
64 bytes from slave1 (192.168.3.66): icmp_seq=2 ttl=64 time=0.476 ms
64 bytes from slave1 (192.168.3.66): icmp_seq=3 ttl=64 time=0.490 ms
64 bytes from slave1 (192.168.3.66): icmp_seq=4 ttl=64 time=0.528 ms
^C
--- slave1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.476/0.520/0.589/0.051 ms
跳转顶部
ssh免密登录
执行命令ssh-keygen -t rsa -P ''
,然后回车即可
[root@host-192-168-3-73 ~]# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:MtzaPUXxJdLDLcvwLxQqWH7HQkr20m7rUgL48IF9f/U root@host-192-168-3-73
The key's randomart image is:
+---[RSA 2048]----+
| oo...|
| + o+*o.|
| + * =.*.= |
| .+.= *.* B .|
| ++S+ *.+ o.|
| =o...= o E|
| . . o+ o . |
| ... |
| o. |
+----[SHA256]-----+
查看/root
下是否有".ssh
"文件夹,且".ssh
"文件下是否有两个刚生产的无密码密钥对。,因为我是使用root
用户来配置的,所以在这目录下,若你使用的时其他用户,则需要在/home/User
目录下寻找.ssh
目录
[root@master .ssh]# pwd
/root/.ssh
[root@master .ssh]# ll
total 8
-rw-------. 1 root root 1675 Mar 16 16:02 id_rsa
-rw-r--r--. 1 root root 393 Mar 16 16:02 id_rsa.pub
将 id_rsa.pub
追加到授权key
文件中
cat id_rsa.pub >> authorized_keys
修改文件权限,若使用的时管理员用户则不需要
chmod 600 authorized_keys
修改SSH
配置文件"/etc/ssh/sshd_config
"的下列内容,需要将该配置字段前面的#
号删除,启用公钥私钥配对认证方式。
PubkeyAuthentication yes
重启服务
systemctl restart sshd
尝试本机嵌套登录,如能不输入密码就表示本机通过密钥登陆验证成功
[root@master .ssh]# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:Hr69gEn5JbaH3pZPvyJ9qhzyCzPYIyleYQyqA+vPz3U.
ECDSA key fingerprint is MD5:f6:f4:9e:7d:c5:b1:8f:68:db:a3:49:66:05:6e:e4:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Last login: Wed Mar 16 15:41:55 2022 from 192.168.0.1
将 Master
节点的公钥 id_rsa.pub
复制到每个 Slave
点,注意不要复制到相同目录下,否则会直接覆盖,建议放到前一个目录
scp id_rsa.pub root@slave1:/root/
[root@master .ssh]# scp id_rsa.pub root@slave1:/root/
The authenticity of host 'slave1 (192.168.0.163)' can't be established.
ECDSA key fingerprint is SHA256:HCyXDBNPToF3n/6WgB/Sj8M9z3IHaGy8CRVTJY6YqQs.
ECDSA key fingerprint is MD5:2e:16:4d:94:00:05:ff:c5:8e:13:08:6a:6a:a9:02:f8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1,192.168.0.163' (ECDSA) to the list of known hosts.
root@slave1's password:
id_rsa.pub 100% 393 314.0KB/s 00:00
在每个Slave
点把 Master
节点复制的公钥复制到 authorized_keys
文件
cat id_rsa.pub >> .ssh/authorized_keys
删除文件
rm -fr id_rsa.pub
将slave1
的公钥发送到master
scp .ssh/id_rsa.pub root@master:/root/
The authenticity of host 'master (192.168.0.162)' can't be established.
ECDSA key fingerprint is SHA256:Hr69gEn5JbaH3pZPvyJ9qhzyCzPYIyleYQyqA+vPz3U.
ECDSA key fingerprint is MD5:f6:f4:9e:7d:c5:b1:8f:68:db:a3:49:66:05:6e:e4:c4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master,192.168.0.162' (ECDSA) to the list of known hosts.
root@master's password:
id_rsa.pub 100% 393 493.3KB/s 00:00
在 Master
节点把从 Slave
节点复制的公钥复制到 authorized_keys
文件
cat id_rsa.pub >> .ssh/authorized_keys
删除文件
rm -fr id_rsa.pub
查看master
节点的authorized_keys
文件,可以发现有两个公钥
[root@master ~]# cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIgiraYbUS+wal7gSzx/kpuZ+ZPnE1Tc+u1QVi25i3ZgoBTOFqjTv973xy3ueExn1udYGmhDDB+vXFxNs2AIgXZEoEpgZAz2kcAEJBjkXT0p8sYXgaliMMFNP8dwiJTCs/YIDol+KIIkIwa3WbQoVEc1zQH1+Xr1Rto1IgLXPRgXO3IMfmX7nqc2ZMdBt0OaPDf2NtBI3e/QDEa59f6J+ge4r8MPuc9C51MeU6NPr20A99Psy1Jbvrr7/Fb2pLxnfne50+4DYjsGPztOgHuQFWoAQ+LDUW6Xhbs5Ig8bUEHt1AILwyNwagJvcsGIvp3wOQt+HRHxJCoAjgPeFsFwJF root@master
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCoPA9uCf/PmUgbpmbPF13VvIwJiSHqAVIpffylbk2g+mEJQMnLxmYv4AVsdc3Wjul2rUMoQh4RPeMFFFincrYFN88DA6SF0F9ZNQOy+6p7CWxLd24hrsn7J69Pab0HxIlMAng8zKjAxZKAOBWyih1nJzqf3UHNdAeZkoe8MbNf6jTXM67vGa0V0FUFU/GvX6st8fLDbROKB8kh1N2X/qLNFiDgxY3Vm1rgN4cDGhs/UqugOHgwnvUScUkjoDQyGn/vYfgHxThHoF+Dv57Xa+bjyUbMmIQYgH7xR/V25F3iU6no3P0LmWsVc4uTTZwdcsPpxMcAfDFL+u5cnivtKrdj root@slave1
这时可以从master
节点来登录slave
[root@master ~]# ssh slave1
Last login: Tue Mar 15 12:18:56 2022 from ::1
[root@slave1 ~]#
需要将每一个节点的公钥都交换
跳转顶部
JDK的配置
将文件解压到指定目录:tar -zxvf jdk-8u162-linux-x64.tar.gz -C /home/
重命名mv jdk1.8.0_162/ jdk
配置环境变量:vi /etc/profile
export JAVA_HOME=/home/jdk
export PATH=$PATH:$JAVA_HOME/bin
使其立即生效
source /etc/profile
测试
[root@host-192-168-3-73 home]# java -version
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
将jdk转发
scp -r /home/jdk slave1:/home
scp -r /home/jdk slave2:/home
[root@host-192-168-3-73 home]# scp /etc/profile slave1:/etc/
profile 100% 1880 1.2MB/s 00:00
[root@host-192-168-3-73 home]# scp /etc/profile slave2:/etc/
profile 100% 1880 1.2MB/s 00:00
跳转顶部
ZooKeeper配置
解压缩:tar -zxvf zookeeper-3.4.5.tar.gz -C /home/
重命名:mv zookeeper-3.4.5/ zookeeper
在根目录下创建两个文件夹
mkdir logs
mkdir data
加入conf目录,生成文件:cp zoo_sample.cfg zoo.cfg
并且修改文件
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/zookeeper/data
dataLogDir=/home/zookeeper/logs
clientPort=2181
server.1=master:2888:3888
server.2=slave1:2889:3889
server.3=slave2:2890:3890
分发
scp -r /home/zookeeper/ slave1:/home/
scp -r /home/zookeeper/ slave2:/home/
在每个节点的zookeeper/data
目录下写入
echo '1' > data/myid
echo '2' > data/myid
echo '3' > data/myid
启动:bin/zkServer.sh start
查看状态
[root@host-192-168-3-73 zookeeper]# bin/zkServer.sh status
JMX enabled by default
Using config: /home/zookeeper/bin/../conf/zoo.cfg
Mode: follower
跳转顶部
Hadoop HA配置文件
解压缩:tar -zxvf hadoop-2.7.7.tar.gz -C /home/
创建目录
mkdir logs
mkdir tmp
mkdir -p dfs/name
mkdir -p dfs/data
修改slaves文件
slave1
slave2
修改hadoop-env.sh文件
export JAVA_HOME=/home/jdk
core-site.xml
<property><name>fs.defaultFS</name><value>hdfs://mycluster</value>
</property><property><name>hadoop.tmp.dir</name><value>/home/hadoop/tmp</value>
</property><property><name>ha.zookeeper.quorum</name><value>master,slave1,slave2</value>
</property><!-- hadoop 链接 zookeeper 的超时时长设置 -->
<property><name>ha.zookeeper.session-timeout.ms</name><value>30000</value>
</property><property><name>fs.trash.interval</name><value>1440</value>
</property>
跳转顶部
hdfs-site.xml
<property><name>dfs.replication</name><value>1</value>
</property><property><name>dfs.namenode.name.dir</name><value>/home/hadoop/dfs/name</value>
</property><property><name>dfs.datanode.data.dir</name><value>/home/hadoop/dfs/data</value>
</property><property><name>dfs.nameservices</name><value>mycluster</value>
</property><property><name>dfs.ha.namenodes.mycluster</name><value>n1,n2</value>
</property><property><name>dfs.namenode.rpc-address.mycluster.n1</name><value>master:8020</value>
</property><property><name>dfs.namenode.rpc-address.mycluster.n2</name><value>slave1:8020</value>
</property><property><name>dfs.namenode.http-address.mycluster.n1</name><value>master:50070</value>
</property><property><name>dfs.namenode.http-address.mycluster.n2</name><value>slave1:50070</value>
</property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://master:8485;slave1:8485;slave2:8485/mycluster</value>
</property><property><name>dfs.journalnode.edits.dir</name><value>/home/hadoop/jn</value>
</property><property><name>dfs.client.failover.proxy.provider.mycluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property><property><name>dfs.ha.fencing.methods</name><value>
sshfence
shell(/bin/true)</value>
</property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value>
</property><property><name>dfs.ha.automatic-failover.enabled.mycluster</name><value>true</value>
</property><!-- 配置 sshfence 隔离机制超时时间 -->
<property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value>
</property><property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>60000</value>
</property><!-- journalnode 集群之间通信的超时时间 -->
<property><name>dfs.qjournal.start-segment.timeout.ms</name><value>60000</value>
</property>
跳转顶部
mapred-site.xml
<!-- 指定 mr 框架为 yarn 方式 -->
<property><name>mapreduce.framework.name</name><value>yarn</value>
</property><!-- 指定 mapreduce jobhistory 地址 -->
<property><name>mapreduce.jobhistory.address</name><value>master:10020</value>
</property><!-- 任务历史服务器的 web 地址 -->
<property><name>mapreduce.jobhistory.webapp.address</name><value>master:19888</value>
</property>
跳转顶部
yarn-site.xml
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>
</property><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value>
</property><property><name>yarn.resourcemanager.cluster-id</name><value>cluster1</value>
</property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>r1,r2</value>
</property><property><name>yarn.resourcemanager.hostname.r1</name><value>master</value>
</property><property><name>yarn.resourcemanager.hostname.r2</name><value>slave1</value>
</property><property><name>yarn.resourcemanager.webapp.address.r1</name><value>master:8088</value>
</property><property><name>yarn.resourcemanager.webapp.address.r2</name><value>slave1:8088</value>
</property><property><name>yarn.resourcemanager.zk-address</name><value>master,slave1,slave2</value>
</property><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value>
</property><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value>
</property>
跳转顶部
启动与测试
分发
scp -r /home/hadoop slave1:/home
scp -r /home/hadoop slave2:/home
每个节点需要先启动journalnode
sbin/hadoop-daemon.sh start journalnode
格式化namenode
bin/hadoop namenode -format
将已格式化的namenode
tmp目录传给另一个namenode
[root@host-192-168-3-73 hadoop]# scp -r dfs/ slave1:/home/hadoop/
VERSION 100% 205 130.7KB/s 00:00
seen_txid 100% 2 1.6KB/s 00:00
fsimage_0000000000000000000.md5 100% 62 68.6KB/s 00:00
fsimage_0000000000000000000 100% 321 270.9KB/s 00:00
或者使用命令:bin/namenode -bootstrapStandby
格式化zkfc
bin/hdfs zkfc -formatZK
在每个namenode
节点上启动zkfc
sbin/hadoop-daemon.sh start zkfc
启动集群
sbin/start-all.sh
查看集群的进程
# namenode1
2752 NameNode
2326 JournalNode
2568 DFSZKFailoverController
1897 QuorumPeerMain
3035 ResourceManager
3182 Jps# namenode2
2080 NodeManager
2194 Jps
1763 JournalNode
1861 DFSZKFailoverController
1656 QuorumPeerMain
1992 DataNode
1933 NameNode# datanode
1953 NodeManager
1866 DataNode
1659 QuorumPeerMain
2107 Jps
1773 JournalNode
查看webui
杀死活跃的主节点
kill -9 2752
一些命令
# 强制切换成等待
bin/hdfs haadmin -transitionToStandby -forcemanual nn1
# 强制切换成活跃
bin/hdfs haadmin -transitionToActive -forcemanual nn1
跳转顶部
【Hadoop HA】搭建Hadoop HA的详细教程相关推荐
- Hadoop 2.0集群配置详细教程
Hadoop 2.0集群配置详细教程 前言 Hadoop2.0介绍 Hadoop是 apache 的开源 项目,开发的主要目的是为了构建可靠,可拓展 scalable ,分布式的系 统, hadoop ...
- Hadoop组件搭建-Hadoop伪分布式
该文章仅用于记录个人笔记,如果您觉得不专业或者有误导,打扰到您,请谅解. Hadoop组件搭建-前置要求 Hadoop组件搭建-jdk Hadoop组件搭建-Hadoop伪分布式 Hadoop组件搭建 ...
- 从购买服务器到搭建WordPress博客详细教程
我的博客原文链接:从购买服务器到搭建WordPress博客详细教程 - Hoyerの小站 〇. 引言 你是否想过搭建一个自己的网站?是否想过随心所欲地设计网站的界面风格?是否想过用自己建的网站赚钱?这 ...
- mac搭建svn服务器文件被锁定,mac安装svn_在MAC下搭建SVN服务器的详细教程
摘要 腾兴网为您分享:在MAC下搭建SVN服务器的详细教程,易信,无忧兼职,茄子快传,每日一淘等软件知识,以及android解锁大师,workflow,乐客联盟,海鸥旅游,mm直播app,腾讯新闻,中 ...
- 最新hexo+github搭建个人博客详细教程
最新hexo+github搭建个人博客详细教程 这是作者自己利用hexo+github搭建博客的完整过程,对自己搭建过程中踩过的坑做一个简单记录. 本人博客最终展示效果:https://huanyay ...
- Hadoop组件搭建-Hadoop HA高可用
在操作之前请先确保java成功安装 并且把前置要求完成 安装HA需要zookeeper组件 请先安装好Zookeeper 组件 解压Hadoop 将hadoop-2.6.0.tar.gz解压到root ...
- Hadoop(二)搭建Hadoop集群
在了解Hadoo基础之后,借助于docker快速实现虚拟化,搭建Hadoop集群,进一步了解Hadoop,实地测试,实地操作,更进一步的了解Hadoop. 首先需要了解Hadoop的各个模块概念已经功 ...
- 搭建个人博客详细教程
基于gitee+hexo搭建个人博客 由于国外的github page访问总是非常慢的,本文选择国内开源代码托管平台码云,因而可以在国内搭建访问与SEO检索都优于GitHub的个人网站.自己亲手勉强搭 ...
- Windows下使用GitHub Pages搭建hexo博客详细教程以及Next主题超全配置
根据下面的教程搭建的博客见这里 前言 马上要进入研究生了,不能再过浑浑噩噩的日子了,搭建一个小网站,用于记录自己所学的东西.其实,之前在腾讯云1元乞丐版上也搭建过博客,但是呢,只用来扯淡了.这次使用 ...
- teamspeak搭建_TeamSpeak服务器配置的详细教程
TeamSpeak是网游语音通信平台,今天小编跟大家分享一篇TeamSpeak服务器配置的详细教程,感兴趣的朋友跟小编一起来了解一下吧! 在Windows下配置一个TeamSpeak服务器仅仅需要安装 ...
最新文章
- ClassCastException
- 基于Python分析深圳程序员工资有多高?
- MyBatis-23MyBatis缓存配置【二级缓存】
- Gradle语法基础解析
- Packet for query is too large(mysql写入数据过大)
- package.json---入门说明
- Asp.NET Core一个接口的多个实现如何基于当前HTTP请求注册
- java get key_java如何获取String里面的键值对:key=valuekey=value
- java 静态线程_Java线程类静态本机void yield()方法(带示例)
- JavaScript 的函数式编程与面向对象编程区别在哪?
- dsp广告和信息流广告区别_信息流广告和sem广告有哪些区别2021?
- oracle 修索引改空间_oracle 修改索引现有表空间
- C++序列式容器(STL序列式容器)
- 递归的经典例子 java_java递归算法经典实例
- 计算机设备维修与及日常保养,电脑主机日常的维护保养计划
- 四分位数计算方法总结
- 交大家简单又好吃的蛋黄酥的做法
- 纽约州立大学石溪分校计算机专业排名,纽约州立大学石溪分校美国大学排名及专业排名汇总(USNEWS美国大学排名版)...
- 同济大学C++mooc第12讲:文件
- Double计算精度丢失(金融入门知识点)