hadoop NameNode HA 和ResouceManager HA

官网配置地址：

HDFS HA : http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

ResourceManager HA : http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

安装jdk

关闭防火墙

hadoop自动HA借助于zookeeper实现，整体架构如下：

m2和m3作为NameNode节点应该配置与其他所有节点的SSH无密码登录

m4和m5应该与m6、m7、m8配置SSH无密码登录

core-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property><name>fs.defaultFS</name><value>hdfs://cluster</value>
</property><property><name>hadoop.tmp.dir</name><value>/home/hadoop/app/hadoop-2.7.3/tmp/data</value>
</property><property><name>ha.zookeeper.quorum</name><value>m6:2181,m7:2181,m8:2181</value></property>
</configuration>

hdfs-site.xml具体配置

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<property><name>dfs.nameservices</name><value>cluster</value>
</property><property><name>dfs.ha.namenodes.cluster</name><value>nn1,nn2</value>
</property><property><name>dfs.namenode.rpc-address.cluster.nn1</name><value>m2:9820</value>
</property>
<property><name>dfs.namenode.rpc-address.cluster.nn2</name><value>m3:9820</value>
</property><property><name>dfs.namenode.http-address.cluster.nn1</name><value>m2:9870</value>
</property>
<property><name>dfs.namenode.http-address.cluster.nn2</name><value>m3:9870</value>
</property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://m6:8485;m7:8485;m8:8485;/cluster</value>
</property><property><name>dfs.client.failover.proxy.provider.cluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property><property><name>dfs.ha.fencing.methods</name><value>sshfenceshell(shell(/bin/true))</value>
</property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hadoop/.ssh/id_rsa</value>
</property><property><name>dfs.journalnode.edits.dir</name><value>/home/hadoop/app/hadoop-2.7.3/journalnode/data</value>
</property><property><name>dfs.replication</name><value>3</value>
</property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property></configuration>

yarn-site.xml具体配置

<?xml version="1.0"?><configuration><!-- Site specific YARN configuration properties --><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value>
</property>
<property><name>yarn.resourcemanager.cluster-id</name><value>cluster</value>
</property>
<property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value>
</property>
<property><name>yarn.resourcemanager.hostname.rm1</name><value>m4</value>
</property>
<property><name>yarn.resourcemanager.hostname.rm2</name><value>m5</value>
</property>
<property><name>yarn.resourcemanager.webapp.address.rm1</name><value>m4:8088</value>
</property>
<property><name>yarn.resourcemanager.webapp.address.rm2</name><value>m5:8088</value>
</property>
<property><name>yarn.resourcemanager.zk-address</name><value>m6:2181,m7:2181,m8:2181</value>
</property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>
</property></configuration>

mapred-site.xml具体配置

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<property><name>mapreduce.framework.name</name><value>yarn</value>
</property>
</configuration>

slaves具体配置

m6
m7
m8

拷贝hadoop到m3、m4、m5、m6、m7、m8

scp -r hadoop-2.7.3/ m3:/home/hadoop/app/
scp -r hadoop-2.7.3/ m4:/home/hadoop/app/
scp -r hadoop-2.7.3/ m5:/home/hadoop/app/
scp -r hadoop-2.7.3/ m6:/home/hadoop/app/
scp -r hadoop-2.7.3/ m7:/home/hadoop/app/
scp -r hadoop-2.7.3/ m8:/home/hadoop/app/

zookeeper配置zoo.cfg（m6 m7 m8）　

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home/hadoop/app/zookeeper-3.3.6/data
# the port at which the clients will connect
clientPort=2181
server.1=m6:2888:3888
server.2=m7:2888:3888
server.3=m8:2888:3888

配置好后的启动顺序：

1、启动zookeeper ./bin/zkServer.sh start

2、分别在m6 m7 m8上启动journalnode， ./hadoop-daemon.sh start journalnode只有第一次才需要手动启动journalnode，以后启动hdfs的时候会自动启动journalnode

3、在m2上格式化namenode，格式化成功后拷贝元数据到m3节点上

4、格式化zkfc ./bin/hdfs zkfc -formatZK 只需要一次

5、启动hdfs

6、启动yarn

验证：

通过kill命令杀死namenode进程观察namenode节点是否会自动切换

yarn rmadmin -getServiceState rm1查看那个resourceManager是active那个是standby

单独启动namenode: ./sbin/hadoop-daemon.sh start namenode

转载于:https://www.cnblogs.com/heml/p/5997190.html