Hadoop Yarn 安装

环境：Linux, 8G 内存，60G 硬盘， Hadoop 2.2.0

为了构建基于Yarn体系的Spark集群，先要安装Hadoop集群，为了以后查阅方便记录了我本次安装的具体步骤。

事前准备

1. 机器准备

三台主机,#后面说明了用途

192.168.1.1 #hadoop1 : master
192.168.1.2 #hadoop2 : datanode1
192.168.1.3 #hadoop3: datanode2

在hadoop1上， vi/etc/sysconfig/network,修改HOSTNAME=hadoop1

在hadoop2上， vi/etc/sysconfig/network,修改HOSTNAME=hadoop2

在hadoop3上， vi/etc/sysconfig/network,修改HOSTNAME=hadoop3

在三台机器上，在/etc/hosts末尾添加

192.168.1.1 hadoop1
192.168.1.2 hadoop2
192.168.1.3 hadoop3

在hadoop1上，运行 hostname hadoop1

在hadoop2上，运行 hostname hadoop2

在hadoop3上，运行 hostname hadoop3

exit重连之后，hostname 就会变成hadoop[1-3],这样做的好处是ssh hadoop2 会自动解析连接192.168.1.2，方便以后使用。这也是短域名实现的方式。

2. 目录创建

$mkdir -p /hadoop/hdfs
$mkdir -p /hadoop/tmp
$mkdir -p /hadoop/log
$mkdir -p /usr/java                               ###java安装路径
$mkdir -p /usr/hadoop                             ###hadoop安装路径
$chmod -R 777 /hadoop

可以根据自己的情况确定安装路径。

安装Java

1. 下载JDK，并安装，建议安装JDK 1.7。本次下载jdk-7u60-linux-x64.tar.gz

http://www.oracle.com/technetwork/java/javase/downloads/index.html

$tar -zxvf jdk-7u60-linux-x64.tar.gz
$mv jdk1.7.0_60 java

注释：下载的java 包类型不同，安装略有不同。

2. 配置Java 环境

可以修改/etc/profile,也可以修改自己home目录下的_{/.profile(ksh)或者}/.bash_profile(bash)，本次安装是bash，所以在.bash_profile 末尾添加

export JAVA_HOME=/usr/java

export CLASSPATH=.:JAVA_HOME/lib/tools.jar:JAVA\_HOME/lib/tools.jar:JAVA_HOME/lib/tools.jar:JAVA_HOME/lib/dt.jar

export PATH=JAVA_HOME/bin:JAVA\_HOME/bin:JAVA_HOME/bin:PATH

使环境立即生效，执行

$source .bash_profile

3. 检查Java是否安装成功

$ java -version
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)

配置SSH 无密码登录

hadoop1 上

$ mkdir .ssh
$ cd .ssh
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/zilzhang/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
The key fingerprint is:
b0:76:89:6a:44:8b:cd:fc:23:a4:3f:69:55:3f:83:e3 ...
$ ls -lrt
total 2
-rw-------   1     887 Jun 30 02:10 id_rsa
-rw-r--r--   1     232 Jun 30 02:10 id_rsa.pub
$ touch authorized_keys
$ cat id_rsa.pub >> authorized_keys

hadoop2和hadoop3上，同样生成公钥和私钥。

[hadoop2]$ mv id_rsa.pub pub2
[hadoop3]$ mv id_rsa.pub pub3

把pub2,pub3都scp到hadoop1上,然后

$ cat pub2 >> authorized_keys
$ cat pub3 >> authorized_keys

把authorized_keys scp到hadoop2和hadoop3上，这样就可以免密码登录了。

一言以蔽之，就是在每台node上生成公钥和私钥，把所有公钥的内容汇总成authorized_keys，并把authorized_keys分发到集群所有node上相同的目录，这样每个node都拥有整个集群node的公钥，互相之间就可以免密码登录了。

验证免密码登录，在hadoop1上：

$ ssh haoop1
ssh: Could not resolve hostname haoop1: Name or service not known
[zilzhang@hadoop3 hadoop]$ ssh hadoop1
The authenticity of host 'hadoop1 (192.168.1.1)' can't be established.
RSA key fingerprint is 18:85:c6:50:0c:15:36:9c:55:34:d7:ab:0e:1c:c7:0f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop1' (RSA) to the list of known hosts.##################################################################                                                               ##       This system is for the use of authorized users only.    ##       Individuals using this computer system without          ##       authority, or in excess of their authority, are         ##       subject to having all of their activities on this       ##       system monitored and recorded by system personnel.      ##                                                               ##       In the course of monitoring individuals improperly      ##       using this system, or in the course of system           ##       maintenance, the activities of authorized users         ##       may also be monitored.                                  ##                                                               ##       Anyone using this system expressly consents to such     ##       monitoring and is advised that if such monitoring       ##       reveals possible evidence of criminal activity,         ##       system personnel may provide the evidence of such       ##       monitoring to law enforcement officials.                ##                                                               ##       This system/database contains restricted data.          ##                                                               ##################################################################[hadoop1 ~]$

安装Hadoop

1. 下载与解压(所有节点)

$ wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz
$ tar -zxvf hadoop-2.2.0.tar.gz
$ mv hadoop-2.2.0 /usr/hadoop

以下都运行在haoop1上

2. 配置环境变量，在.bash_profile末尾添加

export HADOOP_HOME=/usr/hadoop
export HADOOP_MAPARED_HOME=HADOOP_HOMEexportHADOOP_COMMON_HOME={HADOOP\_HOME} export HADOOP\_COMMON\_HOME=HADOOP_HOMEexportHADOOP_COMMON_HOME={HADOOP_HOME}
export HADOOP_HDFS_HOME=HADOOP_HOMEexportYARN_HOME={HADOOP\_HOME} export YARN\_HOME=HADOOP_HOMEexportYARN_HOME={HADOOP_HOME}
export HADOOP_CONF_DIR=HADOOP_HOME/etc/hadoopexportHDFS_CONF_DIR={HADOOP\_HOME}/etc/hadoop export HDFS\_CONF\_DIR=HADOOP_HOME/etc/hadoopexportHDFS_CONF_DIR={HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=HADOOP_HOME/etc/hadoopexportPATH={HADOOP\_HOME}/etc/hadoop export PATH=HADOOP_HOME/etc/hadoopexportPATH=HADOOP_HOME/bin:HADOOP_HOME/sbin:HADOOP\_HOME/sbin:HADOOP_HOME/sbin:PATH

source .bash_profile

3. $HADOOP_HOME/etc/hadoop/hadoop-env.sh,末尾添加

export JAVA_HOME=/usr/java/java

4.$HADOOP_HOME/etc/hadoop/ core-site.xml 添加

hadoop.tmp.dir /hadoop/tmp A base for other temporary directories. fs.default.name hdfs://192.168.1.1:9000

5.$HADOOP_HOME/etc/hadoop/slaves 内容变为（datanode)

192.168.1.2

192.168.1.3

6.$HADOOP_HOME/etc/hadoop/hdfs-site.xml 添加

dfs.replication 3 dfs.namenode.name.dir file:/hadoop/hdfs/name true dfs.federation.nameservice.id ns1 dfs.namenode.backup.address.ns1 192.168.1.1:50100 dfs.namenode.backup.http-address.ns1 192.168.1.1:50105 dfs.federation.nameservices ns1 dfs.namenode.rpc-address.ns1 192.168.1.1:9000 dfs.namenode.rpc-address.ns2 192.168.1.1:9000 dfs.namenode.http-address.ns1 192.168.1.1:23001 dfs.namenode.http-address.ns2 192.168.1.1:13001 dfs.dataname.data.dir file:/hadoop/hdfs/data true dfs.namenode.secondary.http-address.ns1 192.168.1.1:23002 dfs.namenode.secondary.http-address.ns2 192.168.1.1:23002 dfs.namenode.secondary.http-address.ns1 192.168.1.1:23003 dfs.namenode.secondary.http-address.ns2 192.168.1.1:23003

7.$HADOOP_HOME/etc/hadoop/yarn-site.xml添加

yarn.resourcemanager.address 192.168.1.1:18040 yarn.resourcemanager.scheduler.address 192.168.1.1:18030 yarn.resourcemanager.webapp.address 192.168.1.1:50030 yarn.resourcemanager.resource-tracker.address 192.168.1.1:18025 yarn.resourcemanager.admin.address 192.168.1.1:18141 yarn.nodemanager.aux-services **mapreduce\_shuffle** yarn.web-proxy.address hadoop1-9014.lvs01.dev.ebayc3.com:54315

8.$HADOOP_HOME/etc/hadoop/httpfs-site.xml添加

hadoop.proxyuser.root.hosts 192.168.1.1 hadoop.proxyuser.root.groups \*

9.$HADOOP_HOME/etc/hadoop/mapred-site.xml添加(配置job提交到yarn上并且配置history log 服务器)

mapreduce.framework.name yarn Execution framework set to Hadoop YARN. mapreduce.jobhistory.address hadoop1-9014.lvs01.dev.ebayc3.com:10020 mapreduce.jobhistory.webapp.address hadoop1-9014.lvs01.dev.ebayc3.com:19888 mapreduce.jobhistory.intermediate-done-dir /log/tmp mapreduce.jobhistory.done-dir /log/history

这个是说明把job放到yarn 上去跑。

10. 配置同步到其他datanode上

$ scp ~/.bash_profile hadoop2:~/.bash_profile
$ scp $HADOOP_HOME/etc/hadoop/hadoop-env.sh hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/core-site.xml hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/slaves hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/hdfs-site.xml hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/yarn-site.xml hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/httpfs-site.xml hadoop2:$HADOOP_HOME/etc/hadoop/
$ scp $HADOOP_HOME/etc/hadoop/mapred-site.xml hadoop2:$HADOOP_HOME/etc/hadoop/

把hadoop2改成hadoop3,，把配置同步到hadoop3上

启动Hadoop集群

1. 格式化

hadoop namenode -format

2. 启动hdfs

start-dfs.sh

3. 启动yarn

start-yarn.sh

4. 启动history server

mr-jobhistory-daemon.sh start historyserver

5. 启动proxy server

yarn-daemons.sh start proxyserver

httpfs.sh start

6. 创建日志存放目录

hadoop fs -mkdir -p/log/tmp

hadoop fs -mkdir -p/log/history

测试hadoop集群

1. hadoop1,看看进程是否已经开启

$ jps
8606 NameNode
4640 Bootstrap
17007 Jps
16077 ResourceManager
8781 SecondaryNameNode

这些进程必须都有
2. 在hadoop2 上看进程是否开启

$ jps
5992 Jps
5422 NodeManager
3292 DataNode

这些进程必须都有
3. hadoop fs -ls / 看是否可以列出文件

4. 测试hadoop job

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output7

如果运行正常，可以在job monitor页面看到job运行状况。

总结：在安装过程中会遇到各种问题，这里不一一列示，以免太过啰嗦。